1

User Interfaces at Microsoft Research

Intelligent Information Access using Intelligent Information Access using

Animated 2 and 3D Information Animated 2 and 3D Information

VisualizationVisualization

Intelligent Information Access using Intelligent Information Access using

Animated 2 and 3D Information Animated 2 and 3D Information

VisualizationVisualization

Mary Czerwinski

2

A Sweet Spot? Use 3D VisualsUse 3D Visuals

Apply 3D cuesApply 3D cues

Occlusion, Relative Size, ShadowsOcclusion, Relative Size, Shadows

Spatialized AudioSpatialized Audio

2D Interaction technique2D Interaction technique

Little or no egocentric navigationLittle or no egocentric navigation

Objects come to the userObjects come to the user

Use 3D VisualsUse 3D Visuals

Apply 3D cuesApply 3D cues

Occlusion, Relative Size, ShadowsOcclusion, Relative Size, Shadows

Spatialized AudioSpatialized Audio

2D Interaction technique2D Interaction technique

Little or no egocentric navigationLittle or no egocentric navigation

Objects come to the userObjects come to the user

3

Data Mountain

Subject Layout of 100 Pages

“Strongest cue ...relative size”

4

Video

Data MountainData MountainData MountainData Mountain

5

Data Mountain Usability

Study #1 (Compare with IE4 Favorites)Study #1 (Compare with IE4 Favorites)

Reliably faster (26%)Reliably faster (26%)

Study #2 (Longevity and Thumbnails)Study #2 (Longevity and Thumbnails)

After 6 months, no performance changeAfter 6 months, no performance change

Images help, but are not requiredImages help, but are not required

Study #1 (Compare with IE4 Favorites)Study #1 (Compare with IE4 Favorites)

Reliably faster (26%)Reliably faster (26%)

Study #2 (Longevity and Thumbnails)Study #2 (Longevity and Thumbnails)

After 6 months, no performance changeAfter 6 months, no performance change

Images help, but are not requiredImages help, but are not required

6

“Thumbless” Data Mountain

Figure 2: Same Data Mountain, without thumbnails.

7

Figure 3. Average retrieval times to find 100 web pages.

1st vs 2nd visit vs 2nd visit-no images:RT

0

5

10

15

20

25

30

Thumbnail Title Summary All

Retrieval Cue

Re

acti

on

Tim

e (

se

c) 1st visit

2nd visit

2nd visit, no images

Study #2 Results

8

Figure 6. Average retrieval time by thumbnailcondition and trial block (1 block=10 trials).

Thumbnail Condition by Block

0

5

10

15

20

25

30

35

1 2 3 4 5

Block

Avg.

RT

(sec

onds

)

Thumbnails

No Thumbnails

Study #2 Results

9

Visualizing Implicit Queries 2 studies:2 studies:

on managementon management

on retrievalon retrieval

Visualized with DMVisualized with DM

3D, spatial layout 3D, spatial layout of web pagesof web pages

Pages similar to Pages similar to selected page are selected page are highlightedhighlighted

2 studies:2 studies:

on managementon management

on retrievalon retrieval

Visualized with DMVisualized with DM

3D, spatial layout 3D, spatial layout of web pagesof web pages

Pages similar to Pages similar to selected page are selected page are highlightedhighlighted

10

Implicit Queries (IQ): Performance Benefits

IQ users (IQ1 & 2) IQ users (IQ1 & 2)

built more categoriesbuilt more categories

took longer organizingtook longer organizing

faster finding pagesfaster finding pages

Retrieval far more Retrieval far more frequent implies frequent implies IQ faster overallIQ faster overall

IQ users (IQ1 & 2) IQ users (IQ1 & 2)

built more categoriesbuilt more categories

took longer organizingtook longer organizing

faster finding pagesfaster finding pages

Retrieval far more Retrieval far more frequent implies frequent implies IQ faster overallIQ faster overall

Web Page Retrieval Time

0

2

4

6

8

10

12

14

Implicit Query Condition

Ave

rag

e R

T (

sec

on

ds

)

IQ 0

IQ 1

IQ 2

Figure 3. Average retrieval times (includingstandard error of the mean) for the 3 IQ conditions.IQ1,2 use different similarity metrics

11

Glances

Novel 3D navigation techniqueNovel 3D navigation technique

Lightweight, ephemeralLightweight, ephemeral

Two-handedTwo-handed

Non-dominant hand used for glance gestureNon-dominant hand used for glance gesture

Low-cost awareness of surrounding areaLow-cost awareness of surrounding area

Novel 3D navigation techniqueNovel 3D navigation technique

Lightweight, ephemeralLightweight, ephemeral

Two-handedTwo-handed

Non-dominant hand used for glance gestureNon-dominant hand used for glance gesture

Low-cost awareness of surrounding areaLow-cost awareness of surrounding area

12

Toolspaces Objects attached to Objects attached to

virtual bodyvirtual body

Glances used to get Glances used to get to toolspacesto toolspaces

Multiple usesMultiple uses

3D widgets3D widgets

Information storeInformation store

Object transportObject transport

Objects attached to Objects attached to virtual bodyvirtual body

Glances used to get Glances used to get to toolspacesto toolspaces

Multiple usesMultiple uses

3D widgets3D widgets

Information storeInformation store

Object transportObject transport

13

Task Gallery

3D Shell for Windows 3D Shell for Windows

Task ManagementTask Management

Simple, forward-back Simple, forward-back navigationnavigation

Tasks laid out spatially Tasks laid out spatially on floor, ceiling, wallson floor, ceiling, walls

Simple task switchSimple task switch

3D Shell for Windows 3D Shell for Windows

Task ManagementTask Management

Simple, forward-back Simple, forward-back navigationnavigation

Tasks laid out spatially Tasks laid out spatially on floor, ceiling, wallson floor, ceiling, walls

Simple task switchSimple task switch

14

Task Gallery

Simultaneous Simultaneous viewing of viewing of multiple windowsmultiple windows

Simple shift selectSimple shift select

Smart arrangementSmart arrangement

Use 3D to provide Use 3D to provide uniform scalinguniform scaling

Simultaneous Simultaneous viewing of viewing of multiple windowsmultiple windows

Simple shift selectSimple shift select

Smart arrangementSmart arrangement

Use 3D to provide Use 3D to provide uniform scalinguniform scaling

15

Task Gallery Tools Glance Left for Start Glance Left for Start

PalettePalette

Glance Right for Glance Right for Windows ExplorerWindows Explorer

Glance Up or Down Glance Up or Down for status and for status and notificationnotification

Glance Left for Start Glance Left for Start PalettePalette

Glance Right for Glance Right for Windows ExplorerWindows Explorer

Glance Up or Down Glance Up or Down for status and for status and notificationnotification

16

Task Gallery Windows stacked on Windows stacked on

podium or in space until podium or in space until selected for use via selected for use via “flicking”“flicking”

User studies improved User studies improved performanceperformance

Exploring other Exploring other spaces/metaphorsspaces/metaphors

Windows stacked on Windows stacked on podium or in space until podium or in space until selected for use via selected for use via “flicking”“flicking”

User studies improved User studies improved performanceperformance

Exploring other Exploring other spaces/metaphorsspaces/metaphors

17

Video

Task GalleryTask GalleryTask GalleryTask Gallery

18

Hierarchy VisualizationHierarchy Visualization

2D and 3D Research2D and 3D Research

Most frequently requested visualization Most frequently requested visualization solution from MS product teams solution from MS product teams

Hierarchy VisualizationHierarchy Visualization

2D and 3D Research2D and 3D Research

Most frequently requested visualization Most frequently requested visualization solution from MS product teams solution from MS product teams

19

Current Approaches

Windows Tree ControlWindows Tree Control

Many observed problems (from customers and usability Many observed problems (from customers and usability

testing)testing)

Many 2D and 3D hierarchy visualizationsMany 2D and 3D hierarchy visualizations

Each works for some tasks and some scalesEach works for some tasks and some scales

Windows Tree ControlWindows Tree Control

Many observed problems (from customers and usability Many observed problems (from customers and usability

testing)testing)

Many 2D and 3D hierarchy visualizationsMany 2D and 3D hierarchy visualizations

Each works for some tasks and some scalesEach works for some tasks and some scales

20

Problems: Fitting Data

Extreme aspect ratio (broad and shallow)Extreme aspect ratio (broad and shallow)

Balanced tree visualization not informativeBalanced tree visualization not informative

May be multiple hierarchiesMay be multiple hierarchies

Scaling issuesScaling issues

Extreme aspect ratio (broad and shallow)Extreme aspect ratio (broad and shallow)

Balanced tree visualization not informativeBalanced tree visualization not informative

May be multiple hierarchiesMay be multiple hierarchies

Scaling issuesScaling issues

21

Problems: Cognitive Overhead

Loss of context when looking at detailLoss of context when looking at detail

Loss of detail when looking at overviewLoss of detail when looking at overview

Separate detail/overview – extra overheadSeparate detail/overview – extra overhead

Which item is open?Which item is open?

Poor use of display spacePoor use of display space

Loss of context when looking at detailLoss of context when looking at detail

Loss of detail when looking at overviewLoss of detail when looking at overview

Separate detail/overview – extra overheadSeparate detail/overview – extra overhead

Which item is open?Which item is open?

Poor use of display spacePoor use of display space

22

Problems: Fitting User Task

Multiple foci of interest is difficultMultiple foci of interest is difficult

No integration of tree view with searchNo integration of tree view with search

Tree structure may not reflect or support Tree structure may not reflect or support

user task at handuser task at hand

Multiple foci of interest is difficultMultiple foci of interest is difficult

No integration of tree view with searchNo integration of tree view with search

Tree structure may not reflect or support Tree structure may not reflect or support

user task at handuser task at hand

23

Hierarchical Tree Control Problem

24

Basic View Strategies

Two view (separate detail/overview views)Two view (separate detail/overview views)

Distorted viewDistorted view

Distorted data: fisheyeDistorted data: fisheye

Distorted space: 3D, hyperbolicDistorted space: 3D, hyperbolic

Focus in Context (integrated view)Focus in Context (integrated view)

Two view (separate detail/overview views)Two view (separate detail/overview views)

Distorted viewDistorted view

Distorted data: fisheyeDistorted data: fisheye

Distorted space: 3D, hyperbolicDistorted space: 3D, hyperbolic

Focus in Context (integrated view)Focus in Context (integrated view)

25

Basic Visualization Approaches

IndentationIndentation

Tree controlTree control

FisheyeFisheye

ContainmentContainment

TreemapsTreemaps

Pad++Pad++

IndentationIndentation

Tree controlTree control

FisheyeFisheye

ContainmentContainment

TreemapsTreemaps

Pad++Pad++

ClusteringClustering

• Galaxy of NewsGalaxy of News

• ThemeScapeThemeScape

• Hot SauceHot Sauce

GeographicGeographic

• Floor plansFloor plans

• Street mapsStreet maps

26

Basic Visualization ApproachesNode-link diagramsNode-link diagrams

SemNetSemNet

2D diagrams2D diagrams

Cone TreeCone Tree

Fisheye Cone TreeFisheye Cone Tree

Hyperbolic viewerHyperbolic viewer

FSNFSN

XML3DXML3D

Node-link diagramsNode-link diagrams

SemNetSemNet

2D diagrams2D diagrams

Cone TreeCone Tree

Fisheye Cone TreeFisheye Cone Tree

Hyperbolic viewerHyperbolic viewer

FSNFSN

XML3DXML3D

27

Perceptualization GoalsEffective Use of Visual PerceptionEffective Use of Visual Perception

Utilize Visual Perception Characteristics Utilize Visual Perception Characteristics

Encode Documents along Multi-DimensionsEncode Documents along Multi-Dimensions

Add Shading/Transparency Cues to Increase Pre-attentive Add Shading/Transparency Cues to Increase Pre-attentive 3D Perception 3D Perception

Audio PerceptionAudio Perception

Haptic PerceptionHaptic Perception

Effective Use of Visual PerceptionEffective Use of Visual Perception

Utilize Visual Perception Characteristics Utilize Visual Perception Characteristics

Encode Documents along Multi-DimensionsEncode Documents along Multi-Dimensions

Add Shading/Transparency Cues to Increase Pre-attentive Add Shading/Transparency Cues to Increase Pre-attentive 3D Perception 3D Perception

Audio PerceptionAudio Perception

Haptic PerceptionHaptic Perception

28

Perceptive Senses AvailableReady to UseReady to Use

Visual Visual

3D Spatialization 3D Spatialization

Volume / Size Volume / Size

Color Color

Shape (curvature) Shape (curvature)

Opacity Opacity

Texture Texture

Haptic Haptic

Ready to UseReady to Use

Visual Visual

3D Spatialization 3D Spatialization

Volume / Size Volume / Size

Color Color

Shape (curvature) Shape (curvature)

Opacity Opacity

Texture Texture

Haptic Haptic

• Experimental – Proprioception

– Haptic Haptic

– Auditory

– Olfactory

29

XML3D

30

User Study-XML3D•Used Snap.com content; compared XML3D Used Snap.com content; compared XML3D to to www.snap.comwww.snap.com and 2D collapsible tree and 2D collapsible tree control UIcontrol UI

•Target end users were developers and Target end users were developers and maintainers of large web info spacesmaintainers of large web info spaces

•Used search tasksUsed search tasks

•Single or multiple parentsSingle or multiple parents

•New of existing categoriesNew of existing categories

•Used Snap.com content; compared XML3D Used Snap.com content; compared XML3D to to www.snap.comwww.snap.com and 2D collapsible tree and 2D collapsible tree control UIcontrol UI

•Target end users were developers and Target end users were developers and maintainers of large web info spacesmaintainers of large web info spaces

•Used search tasksUsed search tasks

•Single or multiple parentsSingle or multiple parents

•New of existing categoriesNew of existing categories

31

www.snap.com

32

Hierarchical Tree Control UI

33

Task Time Results: XML3D v. 2D UI’s

0

50

100

150

200

250

Existing/Single Existing/Multiple New /Single New /Multiple

Category X Parent Task Condition

Seco

nd

s XML3D

2D

34

Consistency, Tool Usage ResultsFor existing For existing categories, categories, XML3D benefits XML3D benefits do not sacrifice do not sacrifice quality of searchquality of search

Users preferred Users preferred 2D lists for new 2D lists for new category tasks. category tasks. Why?Why?

For existing For existing categories, categories, XML3D benefits XML3D benefits do not sacrifice do not sacrifice quality of searchquality of search

Users preferred Users preferred 2D lists for new 2D lists for new category tasks. category tasks. Why?Why?

XML3D Feature Usage

02468

1012

3D graph List 3D graph List

Existing New

Category

Avg

. # T

imes

Use

d

35

Direction…Using perceptual cues to provide important Using perceptual cues to provide important visual pattern information to uservisual pattern information to user

User can search or browseUser can search or browse

3D final arrangement can emphasize 3D final arrangement can emphasize multiple/user defined attributesmultiple/user defined attributes

Ready to drop in audio, haptic, clustering, Ready to drop in audio, haptic, clustering, implicit queryimplicit query

Using perceptual cues to provide important Using perceptual cues to provide important visual pattern information to uservisual pattern information to user

User can search or browseUser can search or browse

3D final arrangement can emphasize 3D final arrangement can emphasize multiple/user defined attributesmultiple/user defined attributes

Ready to drop in audio, haptic, clustering, Ready to drop in audio, haptic, clustering, implicit queryimplicit query

36

3D UI Conclusions Exploration of Desktop 3D “sweet spots” Exploration of Desktop 3D “sweet spots”

Perform well against existing GUIsPerform well against existing GUIs

Usability test informs design evolutionUsability test informs design evolution

Audio in 3D environment is usefulAudio in 3D environment is useful

Let users choose organizationLet users choose organization

Spatial memory and perceptual cues critical in Spatial memory and perceptual cues critical in Desktop 3DDesktop 3D

Exploration of Desktop 3D “sweet spots” Exploration of Desktop 3D “sweet spots”

Perform well against existing GUIsPerform well against existing GUIs

Usability test informs design evolutionUsability test informs design evolution

Audio in 3D environment is usefulAudio in 3D environment is useful

Let users choose organizationLet users choose organization

Spatial memory and perceptual cues critical in Spatial memory and perceptual cues critical in Desktop 3DDesktop 3D

37

3D User Interfaces Future Work

High value non-game apps:High value non-game apps:

Shell/Web/Mail/File BrowserShell/Web/Mail/File Browser

New 3D interaction techniquesNew 3D interaction techniques

Navigation and manipulationNavigation and manipulation

Input devices/techniquesInput devices/techniques

New visualization techniquesNew visualization techniques

Prove value of 3D and interactive animationProve value of 3D and interactive animation

High value non-game apps:High value non-game apps:

Shell/Web/Mail/File BrowserShell/Web/Mail/File Browser

New 3D interaction techniquesNew 3D interaction techniques

Navigation and manipulationNavigation and manipulation

Input devices/techniquesInput devices/techniques

New visualization techniquesNew visualization techniques

Prove value of 3D and interactive animationProve value of 3D and interactive animation