making sense on

small screens

patrick baudisch

microsoft research

adaptive systems and interaction

interaction focus

small screensstore content off-screen

halo

store content in another layer

multiblending

squeezing it in

fishnetmake the unreadable readable

summary thumbnailsmanually collapse irrelevant material

collapse-to-zoom

(making two small screens into one big

stitching, mouse ether) collapse

expand

return

scenarios…

• commuting between Seattle & Redmond• mobile user interface supports

• recall: bookmarked weather report site• execute: navigation system

• the real work I do on my desktop• PC cell phone sync cable

but not for all…

• PCs: 300 million

• cell phones: 1.5 billion

…the “others”

• rural computing, such as India• phone users, never had access to PC• phone = first access to the digital world

problems

• 1. phone-only usersneed to• plan and understand

maps and routes• explore the web interactively• analyze and compare data

(sensemaking)• but almost all apps designed

for phone+PC combo users

• 2. screen too small• most content designed for desktop PC

[rooms, card et al 86]

research agenda

• long term goalhow much and what functionality can/should we offer phone only users?

• short term goalovercome limitations of small screen size

oh, and, does it matter?

• PCs market• at 300 million PCs fairly saturated

• cell phones• already 5 times bigger• growing at high speed

related work

skip

related work: zooming

• [Xie etc. al, www’04]:tap to zoom into a tile

related work: overview+detail

• [O’Hara et. at CHI 99]: readable text on hover

related work: multiple foci• [Wobbrock

et. al UIST’02]: Web thumb

related work: popouts

• popout prism [suh, et al., chi’02]

• enhanced thumbnails

related work: fisheyes

• document lens [robertson, uist’93]• unifying presentation space [carpendale, uist’01]• focus+context sketching on a Pocket PC [lank, chi’04]

fishnet,collapse,s-thumb

related work: peep hole

• Ka-Ping Yee [CHI’03]

collapse

expand

add off screen

squish context

make readable

collapse irrelevant

add a layer

halo

halo

store content in periphery

the problem

+

halo <demo>

cinematography

1. entry and exit points

2. point of viewarrow-based techniques

3. partially out of the frame halo

rings are familiar, graceful degradation

streetlamps

• aura visible from distance• aura is round• overlapping auras aggregate• fading of aura indicates distance

what we changed• smooth transition sharp edge• disks rings• dark background light background

title: user study

user study

interfacesarc/arrow fading offscale 110-300m/cmmap as backdropreadability oksame selectable

size

hypothesis:

halo faster

halo ring distance from display border

legend

tasks

(a) locate (b) closest

(d) avoid(c) traverse

procedure

• 12 participants• within subject design, counterbalanced• four training maps per interface/task,

then eight timed maps• questionnaire

task completion time

Task Arrow interface Halo interface

Locate 20.1 (7.3) 16.8 (6.7)

Closest 9.9 (10.1) 6.6 (5.3)

Traverse 20.6 (14.1) 16.8 (8.7)

Avoid 9.2 (4.7) 7.7 (5.8)

0

5

10

15

20

25

Locate Closest Traverse Avoid

Arrow interface

Halo interface

33%

16%

error rateTask Arrow interface Halo interface

Locate 23.5 pixels (21.6) 28.4 pixels (33.8)

Closest 22% (42%) 21% (41%)

Traverse 97.4 pixels (94.7) 81.0 pixels (96.7)

Avoid 15% (35%) 14% (34%)

0

5

10

15

20

25

30

Locate Closest Traverse Avoid

Arrow interface

Halo interface

participants underestimated distances by 26% participants saw ovals (gestalt laws?) we can compensate for that: width += 35%

0

1

2

3

4

5

6

7

8

9

Locate Closest Traverse Avoid

Arrow interface

Halo interface

subjective preference

results

• halo 16%-33% faster than arrows• no split attention• distortion-free space• scale independent• no need to annotate distance• perceive all rings at once

[treisman & gormican]

• limitation: max number or rings

> back to large displays

teaser: multiblending

• application UI competes with content for screen space

• where to put it?

> back to large displays

multiblending

collapse

expand

add off screen

squish context

make readable

collapse irrelevant

add a layer

conclude

multiblending: summary

solution multiblending:glass palette distinguished from background photo& background more recognizable

solution multiblending:glass palette distinguished from background photo& background more recognizable

problem with traditional alpha blending: is thisbush in palette or background?

problem with traditional alpha blending: is thisbush in palette or background?

goal of (alpha)-blending

• goal: get twice the benefitfrom limited screen space

• applications of alpha palettes • everquest• but not photoshop

• example• overview: explain detail view• photo: context for editing

limitations of alpha-blending

• limitations• loss of contrast in both layers• colors are not “true” anymore• interference makes it unclear

which layer a feature belongs to

• all features are affected

• reason• each channel (rgb)

= weighted sum of respective channel in source images

multiblending

• preserve the most relevant features

allow different weight for each features class

• avoid visual ambiguity

to preserve feature in one layer eliminate that feature in other layers

use perception-oriented feature classes (CIE Lab color + edges)

glass palette: steps 1-2

1. desaturate 2. emboss…

glass palette: step 3

3. blurbackground…

problems withnoisy background

glass palette: step 4

4. removeunused areas

unuseddecoration

step 5 & limitation

5. …re-map features

collision: layers with same requirements

opaque

alphablending

complete example

multiblending

implementation

• recent graphics cards (with pixel shader 2.0) does computation on the fly

• 145 frames/sec

general procedure

• glass palettes is just one possible example

• other application scenarios may favor different palette styles

• general procedure: for each feature class• decide which layer benefits more from it• eliminate feature from other layer

or map it to a different feature class

user studies

• compare multiblending with alpha blending at several levels of opacity

background visibility

palette

visibility

better technique

background visibility

background visibility

0.23

0.33

0.42

0.25

0.34

0.45 0.46

0.15

0.0

0.1

0.2

0.3

0.4

0.5

0.6

alpha-10 alpha-25 alpha-50 glass

erro

r am

ount

(st

eps)

light featuresdark featureslight featuresdark features

palette visibility

palette visibility

0.38

0.02 0.01 0.01

1.58

0.05 0.02 0.06

0.0

0.5

1.0

1.5

2.0

2.5

alpha-10 alpha-25 alpha-50 glass

erro

rs p

er t

rial

over light backgroundover dark backgroundover light backgroundover dark background

results

• multiblending• is a generalization of alpha blending• provides both background and foreground visibility

simultaneously (no single alpha value can do this)• avoids interference• allows preserving the most relevant features

• is widely applicable• also for desktop: chameleon skin, games (Everquest)…

ok, back to large documents: web pages

ok, back to large documents: web pages

fishnet

collapse

expand

add off screen

make readable

collapse irrelevant

add a layer

conclude

squish context

fishnet

squish context

summary: highlighted search

US

Blackout

PGA

Microsoft

found!

found!

don’t know

don’t know

• highlights tell me “found”• …but cannot tell me “not there”

fishnet

not there!not there!

found!found!

benefit: judge relevanceof web page at a glance

benefit: judge relevanceof web page at a glance

Microso

design

scrolling, context highlighting

space fillingspace filling

use parallel projection

manhattan lens zoomscapes central projection parallel projection

readabilitybottlenecks

contentdisrupted

tall objectswon’t shrink

• two tasks…thus two resolutions• reading: render as much as possible at full size• skimming: preserve readability of context as well as possible

• homogenous context resolution preserves alignment

space filling

context area: popouts

• ensure that search terms are readable

• preserves font attribute, such as underline

• shows anchor

extended highlights

• implementation: place color-full table cell behind

popout prism:popouts interfere(need to turn off to read)

fishnet:similar saliency withextended highlights

anchoring

focus area

top contextcontext

bottom context

anchor points popout

user study

user study

• goal: when to use fisheye view; when not to use it 4 tasks

• 3 interfaces with comparable functionality• linear: standard internet explorer• overview: left 25% of screen = overview• fisheye: top & bottom 25% = context

• size & aspect ration:• same amount of screen space• fit in professional web pages, 800 pixels wide with overview/context: different aspect ratios

• 13 participants (7 males, 6 females)

task 1: In this page outdated? babydisney video sale

task 2:productchoice

512 MB

DVD

XP Home

task 3: co-occurrenceof and Mariners Mets

task 4:textanalysis

Clinton

… and hypotheses

distinguishing columns immaterial …

…necessary

distinguishing rows immaterial

favors overview-fisheye favorsfisheye

… necessaryfavors overview favors linear view

outdated task

product comparison

analysis

co-occurrence task

02468

101214161820

productchoice

co-occur-rence

is this pageout-dated?

analysistas

k c

om

ple

tio

n t

ime

(in

sec

)

Linear Overview Fisheye

results

confirmed hypotheses

subjectivepreference

• per interface + task• consistent with task time• exception: overview preferred

for the co-occurrence task

• however overall• 10 participants preferred overview,

only 3 preferred fisheye, none preferred linear• fisheye view less familiar, more cognitive effort?

02

46810

121416

1820

Out-dated Product choice Co-occurrence Analysis

Tim

e (i

n s

ec)

0

1

2

3

4

5

6

7

8

Out-datedProduct choice Co-occurrence Analysis

Linear Overview Fisheye

task completion time

satisfaction

discussion:when to use what?• fisheye views + search term

highlighting results in useful interface

• there is no black and white wrt fisheye views

• use fisheye view• if grouping in rows does not matter

• use overview• if grouping in columns does not matter

• consider fisheye & overview on demand• unlike overview, fisheye no page reflow

but wait: web pages aretoo wide for PDA

summary thumbnails

collapse

expand

add off screen

squish context

make readable

collapse irrelevant

add a layer

conclude

unreadablereadable

• any screen size• any font size

demo

implementationscale font up to 23pt2

crop text to repair3

render page at small scale4

mars mission

mars mission

mars m

bitmap

html

• load in web browser control

• for each mshtml element indocument object model (DOM)• count lines• increase font size• reduce text to preserve #lines

• client does scaling

text reduction

• current • cut off at the end of paragraph• OR remove common words

• future• use something smarter

(but what we have works surprisingly well)

• however• goal is to show users where to zoom in• may not be a summarizer

related work

user study design

• first set of “pilots” (20 students)• go to history, pick fifth page, tell us URL and task

• second set of “pilots” (2 internal)• pilots to generate balanced sets, remove 1

outlier page

• actual participants (12 externally recruited)• the actual study

results

Thumbnail SummaryThumbnail

Single Column

# o

f p

arti

cip

ants

0

2

4

6

8

10

0

5

10

15

20

25

Thumbnail Semantic Th. Single Column Desktop

Me

an

ta

sk

tim

es

(s

)

task time

preference

more results

0

10

20

30

TN ST SC DT

Tot

al v

ertic

al s

crol

l

0

10

20

30

TN ST SC DT# ve

rt s

crol

l dir

chan

ge

a

c

0

2

4

6

8

10

12

14

Thumbnail Summary Thumbnail

# o

f zo

om

ing

ev

en

ts

# zooming

scrolling

discussion

• summary thumbnails subsume traditional thumbnails

• simple conversion using a proxy server

• any platform

product

collapse-to-zoom

collapse

expand

add off screen

squish context

make readable

add a layer

collapse irrelevant

conclude

collapse-to-zoom

narrow down manually

collapse

expand

short demo

(there will be a second, slightly longer demo at the end of this talk)

?

limitation of thumbnails

• irrelevant page content costs valuable screen space

• when used with traditional thumbnails page content can become unreadable…

related work

related work

• approaches• device-specific authoring

• multiple-device authoring

• automatic re-authoring

• and client-side navigation

thumbnail viewscollapse-to-zoom

collapse-to-zoom

• allow users to use their knowledge about relevant areas zoom in

(arbitrary rectangular regions)

.

collapse-to-zoom

• allow users to use their knowledge about relevant areas zoom in

• …but also allow leveraging their knowledge about what is not relevant collapse

collapse-to-zoom

• always show full page width• use freed space to grow

remaining (relevant) content

collapse-to-zoom

• provide visual context at all times: placeholders

collapse-to-zoom

• allow bookmarking collapsed state

marquee menu:there are 4 ways to select

• today: no distinction between the four ways of opening rectangular selection

marquee menu: direction selects 1 of 4 commands

collapse-cell

expand-cell

collapse-column

expand&zoom

expand

collapse

an

mai

n

com

ds m

the name of the game

• photoshop: “marquee selection”

• “marking menu”:selecting commands with a pen stroke

• combine both marquee menu

transfer to smartphone

• generic• label cells with numeric

codes and let users enter cell label [Paek et al. CSCW 2004]

• simple: reduce to 1D• collapse column and

expand column

webTV

implementation

• runs on desktop / tabletPC• page-splitting based on DOM

• limitation: can’t start drag over link user study

collapse

expand

add off screen

squish context

make readable

collapse irrelevant

add a layer

next steps

integrating it all

collapse-to-zoomfor interaction

summary thumbnailsfits page to screen width

+ +fishnet

fits page to screen height

conclusions

• new visualizations techniquesare the foundation for viewing largedocuments, such as web pages

next steps

• support users’ sensemaking activities on small screen devices• how to pick the best insurance on a PDA?• how to author a web page on a phone?• how to do a tax return on a phone?

read more

collapse

expand

add off screen

squish context

make readable

collapse irrelevant

add a layer

CHI 2003

CHI 2004

UIST 2004

CHI 2005AVI 2004

patrickbaudisch.com/projects

thanks!

• thanks to all the coauthors• ruth rosenholtz• carl gutwin• bongshin lee• heidi lam• xing xie

• and VIBE

END

related work: fisheyes

• works well to reduce page length

• applying it to page width works not so well• forces users to scroll for

each line

fishnet [Baudisch, AVI 2004]

back to collapse-to-zoom or summary thumbnails

next visit of that page:page is already pre-collapsedwhich leads to fullyreadable detail viewUser uses expand-and-zoomgesture…User can follow links directlyfrom within overviewContent area expands moreand is now readable…make page content grown.Now collapses “menu” columnuser collapses“archive” column to…

collapse-to-zoom:

walkthrough

thumbnail view:unreadably small

next:search

1. locate task

click at expected location of off-screen targets

had tosimulate on PC

2. closest task

click arrow/arc or off-screen location closest to car

3. traverse task

4. avoid task

click on hospital farthest away from traffic jams

background visibility

50% gp

25%10%

palette visibility

palette visibility

50% gp25%10%

background visibility

50% gp

25%10%

why even consider fisheye?

• web pages contain little spatial information

• limit distortion to what is absolutely necessary

• enhance periphery with something useful:search term popouts

research question

• for what types of pages does highlighted search benefit from fisheye view?

• different from [hornbæck & frokjær, chi’01] • fishnet shows entire page• integrated navigation• search term highlighting add utility to context

areas

independent variables…

distinguishing columns immaterial …

…necessary

distinguishing rows immaterial

task 1 task 2

… necessary task 3 task 4