Pushing the Boundaries of Interactionin Data Visualization

John Stasko

School of Interactive ComputingGeorgia Institute of Technology

stasko@cc.gatech.edu

September 18, 2019

Information Visualization

Developing representations for data that are not inherently spatial (abstract)

Sports statisticsStock trendsCar attributesConnections between criminals...

2

Applications of VisualizationPresentationAnalysis

(Explanatory)(Exploratory)

3

4

"Whatever"

1. Presentation

Purpose:Communicate data and ideasExplain and informProvide evidence and supportInfluence and persuade

5

Frequent presentation goalsClarifyFocusHighlightPersuade

May just show a few variablesand/or a subset of the data cases

6

7

Presentation Characteristics

"Insights" already known, need to be communicated

Usually static

An Example

https://depictdatastudio.com/how-to-tell-a-story-with-data-titles-subtitles-annotations-dark-light-contrast-and-selective-labeling/

8

Staticinfographics

http://www.ivan.cash/infographic-of-infographics9

http://www.accurat.it10

http://weather-radials.com/

11

http://www.r2d3.us/visual-intro-to-machine-learning-part-1/

Interactivevisualizations

12

http://www.fallen.io/ww2/

13

Datavideos

14

https://vimeo.com/354276689

15

https://www.visualcinnamon.com/

16

17

18

19

20

21

22

Graph-Level Operations (GLOs)

Stolper, et alTVCG (InfoVis) ‘14

What might graph visualizationbuilding blocks look like?

Encapsulated manipulations of graph visualizations

23

Graph Representations

24

Positioning Nodes

1. Align Nodes

2. Evenly Distribute Nodes

3. Evenly Distribute Nodes by Attribute

4. Substrate Nodes by Attribute

5. Evenly Distribute Nodes within Substrates

6. Position Nodes Relatively

7. Evenly Distribute Nodes Radially by Attribute

8. Evenly Distribute Nodes Radially

9. Position Nodes Radially by Attribute

10. Substrate Nodes Radially by Attribute

11. Evenly Distribute Nodes Along Plot Radius

12. Evenly Distribute Nodes Along Plot Radius

13. Position Nodes Along Plot Radius by Attribute

14. Substrate Nodes Along Plot Radius

15. Position Nodes Along Plot Radius by Constant16. Apply an Algorithm to the Nodes

Modifying Element Properties

17. Size Nodes by a Constant

18. Size Nodes Relatively by a Continuous Attribute

19. Display All Links

20. Display Selected Links

21. Hide Links

22. Display Links as Straight

23. Display Links as Curved

24. Display Links as Circles

Cloning Nodes

25. Clone Active Generation

26. Select Generation k

27. Set Source Generation k

28. Set Target Generation k

29. Remove Generation k

Aggregating Nodes and Edges

25. Clone Active Generation

26. Select Generation k

27. Set Source Generation k

28. Set Target Generation k

29. Remove Generation k

30. Aggregate by Attribute

31. Aggregate by Attribute and Attribute

32. Deaggregate Generation k

Modifying Display Properties

33. Show Axis

34. Hide Axis

34 Operations, 5 Categories

25

26

2. Analysis

Purpose:Explore dataAssess a situationDetermine how to proceedDecide what to do

27

Frequent analysis goalsShow many/all variablesIllustrate overview and detailFacilitate comparison

Display may not be easy to interpret at first

28

29

Analysis Characteristics

"Insights" not yet known, need to be found

If data is large enough, usually interactive

Constituents

“The effectiveness of information visualization hinges ontwo things: its ability to clearly and accurately representinformation and our ability to interact with it to figure outwhat the information means.”

S. Few, Now you see it

Two key aspects of data visualizationRepresentationInteraction

30

InteractionHow manifested today?

Chad Stolper

1. Tooltips & selectionGet details

2. Navigation3. Brushing & linking

31

InteractionCan we do more?

Employ interaction in a more fundamental mannerto strengthen the power of visualization

Some examples…

32

33

34

http://iilabgt.org/vibliography

35

TasksWhat is author X's profile?What are the most cited papers?What does this paper cite and who cites it?How have themes changed over the years?How active has organization Y been?

CiteVis

Stasko, Choo, Han, Hu, Pileggi, Sadana & StolperInfoVis poster ‘13

http://www.cc.gatech.edu/gvu/ii/citevis

Demo

36

InteractionWhy interact?

37

1. Select2. Explore3. Reconfigure4. Encode5. Abstract/Elaborate6. Filter7. Connect

Yi, Kang, Stasko & JackoTVCG (InfoVis) ‘07

Intent

InteractionHow do we interact?

Traditional – Desktop: keyboard, mouse

New – Tablet: fingers and multi-touch

ChallengesLimited real estateSmall visual elements to select“Fat finger” problemLack of mouse hover…

38

Moving to Tablets

Sadana & StaskoAVI ’14, EuroVis ‘16

http://www.cc.gatech.edu/gvu/ii/touch/

Tangere

39

Goal: Bring traditional infovischarts to the tablet

40

Start with a scatterplot

41

Integrate with multiple views

42

How about graphs/networks?

43

“A Set of Multi-touch Graph Interaction Techniques”Schmidt, Nacenta, Dachselt, Carpendale

ITS '10

Multi-hand gestures formanipulating edges

44

TangrapheInteractive Exploration of Network Visualizationsusing Single Hand, Multi-touch Gestures

Thompson, Srinivasan, & StaskoAVI '18

Operations:SelectionAdjacency-based explorationLayout modificationNavigation

• Nodes

• Links

• Region

• First-degree network

Selection

• Nodes

• Links

• Region

• First-degree network

Selection

• Nodes

• Links

• Region

• First-degree network

Selection

• Nodes

• Links

• Region

• First-degree network

Selection

• Expand/Collapse

Adjacency-based Exploration

(( ))

• Pinning nodes

• Positioning neighbors

Layout Modification

• Pinning nodes

• Positioning neighbors

Layout Modification

• Pan & Zoom

Navigation

Selection

Adjacency-basedExploration

Navigation

LayoutModification

Single hand Gestures

(( ))

Single hand Gestures

• Enable interaction on mobile devices

Single hand Gestures

• Enable interaction on mobile devices

• Facilitate additional input devices

How about speech & NLI?

Multimodal interaction

Why just touch?

What would people say to a visualization?

56

57

Operations

Find Nodes

Find Connections

Find Path

Filter Nodes

Color Nodes

Size Nodes

Interface Actions

Operations

Find Nodes

Find Connections

Find Path

Filter Nodes

Color Nodes

Size Nodes

Interface Actions

Orkohttp://www.cc.gatech.edu/gvu/ii/naturalvis/

Srinivasan & StaskoTVCG (InfoVis) '1760

Network of European soccer playersEdges: Club or country connection

61

Find Connections

Target

“Show Ronaldo’s connections”

Target

Operation:

Explicit

Contextual& Follow-up

High-level

Find Ronaldo’s connections.Show connections between Pogba and Bale.Highlight the shortest path from Evra to Kroos.Color by position.Size nodes by betweenness centrality.Only show German forwards....

Are any of these players right footed?Filter by this player’s club.Show connections of these players.Color nodes by country > Now club > How about position?Show German strikers with more than 30 goals > How

about French strikers?...

How are France and Italy connected?Players from which countries tend to play more with clubs in

the same country?Find interesting clusters of players.Modify the network to focus on English players....

Explicit

Contextual& Follow-up

High-level

Find Ronaldo’s connections.Show connections between Pogba and Bale.Highlight the shortest path from Evra to Kroos.Color by position.Size nodes by betweenness centrality.Only show German forwards....

Are any of these players right footed?Filter by this player’s club.Show connections of these players.Color nodes by country > Now club > How about position?Show German strikers with more than 30 goals > How

about French strikers?...

How are France and Italy connected?Players from which countries tend to play more with clubs in

the same country?Find interesting clusters of players.Modify the network to focus on English players....

Explicit

Contextual& Follow-up

High-level

Find Ronaldo’s connections.Show connections between Pogba and Bale.Highlight the shortest path from Evra to Kroos.Color by position.Size nodes by betweenness centrality.Only show German forwards....

Are any of these players right footed?Filter by this player’s club.Show connections of these players.Color nodes by country > Now club > How about position?Show German strikers with more than 30 goals > How

about French strikers?...

How are France and Italy connected?Players from which countries tend to play more with clubs in

the same country?Find interesting clusters of players.Modify the network to focus on English players....

Explicit

Contextual& Follow-up

High-level

Find Ronaldo’s connections.Show connections between Pogba and Bale.Highlight the shortest path from Evra to Kroos.Color by position.Size nodes by betweenness centrality.Only show German forwards....

Are any of these players right footed?Filter by this player’s club.Show connections of these players.Color nodes by country > Now club > How about position?Show German strikers with more than 30 goals > How

about French strikers?...

How are France and Italy connected?Players from which countries tend to play more with clubs in

the same country?Find interesting clusters of players.Modify the network to focus on English players....

Show nodes connected to Ronaldo

Show Ronaldo's connections.Find players linked to Ronaldo.Highlight players who play with Ronaldo.Which players play in the same team as Ronaldo?Show nodes directly connected to Ronaldo.Find nodes adjacent to Ronaldo.Show Ronaldo's teammates.Who all is Ronaldo directly connected to?Find players with a direct link to Ronaldo.Find direct connections of Ronaldo.…

But it’s even trickier…

NL Query Processor

Database

ServerClient

Interface Manager

Response Processor

Response Generator

“Show connections of English players with more than 20 goals”

“Show connections of these players”

“Show English players with more than 20 goals”

“Show connections”

Goal: To find connections of high goal scoring players for England

> “Show England players”> “Show players with more than 20 goals”> “Show connections”

• Active/highlighted nodes

• Active filters

• Active visual encodings

• Operations & targets from previous query

Context

(new/ current query)

Ambiguity Widgets

Gao et al., UIST ‘15

Query Manipulation Widgets

Operation Suggestion

Proactive SummaryChart Reordering

User Study

Six participantsNetwork of European soccer playersGiven series of ten questions/tasks to accomplish (no training)~30 minutes

76

77

P1 P2 P3 P4 P5 P6

S T ST S T TS S T ST S T ST S T ST S T ST TS

T1 1 2 1 1 1 1

T2 2 1 1 1 1 1

T3 2 2 1 3 1 1 1 3 1 3 1 2

T4 2 1 3 4 3 6 3

T5 2 2 1 1 1 2 4 4 1 1

T6 1 1 1 2 1 1 1 3 4

T7 1 1 2 3 1 1 1 1 1 3 1 2 2

T8 1 1 1 1 1 1 1 2 1 1

T9 2 2 2 2 2 1 1 2

T10 2 2 2 8 1 2 6 2 2 5 2 5 2 3 1

S: SpeechT: TouchST: Speech+TouchTS: Touch+Speech

Participants

Task

s

78

P1 P2 P3 P4 P5 P6

S T ST S T TS S T ST S T ST S T ST S T ST TS

T1 1 2 1 1 1 1

T2 2 1 1 1 1 1

T3 2 2 1 3 1 1 1 3 1 3 1 2

T4 2 1 3 4 3 6 3

T5 2 2 1 1 1 2 4 4 1 1

T6 1 1 1 2 1 1 1 3 4

T7 1 1 2 3 1 1 1 1 1 3 1 2 2

T8 1 1 1 1 1 1 1 2 1 1

T9 2 2 2 2 2 1 1 2

T10 2 2 2 8 1 2 6 2 2 5 2 5 2 3 1

S: SpeechT: TouchST: Speech+TouchTS: Touch+Speech

Participants

Task

s

Speech (individually) was the dominant input modality (~50%)

79

P1 P2 P3 P4 P5 P6

S T ST S T TS S T ST S T ST S T ST S T ST TS

T1 1 2 1 1 1 1

T2 2 1 1 1 1 1

T3 2 2 1 3 1 1 1 3 1 3 1 2

T4 2 1 3 4 3 6 3

T5 2 2 1 1 1 2 4 4 1 1

T6 1 1 1 2 1 1 1 3 4

T7 1 1 2 3 1 1 1 1 1 3 1 2 2

T8 1 1 1 1 1 1 1 2 1 1

T9 2 2 2 2 2 1 1 2

T10 2 2 2 8 1 2 6 2 2 5 2 5 2 3 1

S: SpeechT: TouchST: Speech+TouchTS: Touch+Speech

Participants

Task

s

Only three instances of sequential input where touch preceded speech

80

P1 P2 P3 P4 P5 P6

S T ST S T TS S T ST S T ST S T ST S T ST TS

T1 1 2 1 1 1 1

T2 2 1 1 1 1 1

T3 2 2 1 3 1 1 1 3 1 3 1 2

T4 2 1 3 4 3 6 3

T5 2 2 1 1 1 2 4 4 1 1

T6 1 1 1 2 1 1 1 3 4

T7 1 1 2 3 1 1 1 1 1 3 1 2 2

T8 1 1 1 1 1 1 1 2 1 1

T9 2 2 2 2 2 1 1 2

T10 2 2 2 8 1 2 6 2 2 5 2 5 2 3 1

S: SpeechT: TouchST: Speech+TouchTS: Touch+Speech

Participants

Task

s

30 instances of sequential input where speech preceded touch

81

● “It [multimodal interaction] was fun to use and a very intuitive way to explore a network”

● “I was surprised by the speech feature. I did not expect it to work as well as it did”

● “...having worked with many visualization programs before, having to go through and manually clicking is really annoying especially when you have a ton of dropdowns. So I really like the speech feature, I know it’s still in a rudimentary stage but it does a really good job”

Positive

● “It was a little frustrating when the system did not understand my voice or did not react at all to voice”

Negative

Qualitative Feedback

82

Potential Future Work

"Hey system, bundle the edges going to Messi"

"Hey system, change to a Sugiyama layout"

Take AwaysThink carefully about purpose: communication -- analysis

83

VS.

Take AwaysThink carefully about purpose: communication -- analysis

Interaction provides power, innovate with it

84

Take AwaysThink carefully about purpose: communication -- analysis

Interaction provides power, innovate with it

Multimodal interaction is coming…

85

Take AwaysThink carefully about purpose: communication -- analysis

Interaction provides power, innovate with it

Multimodal interaction is coming…

86

Thanks to…

87

Acknowledgments

• Supported by NSF IIS-1717111, NSF IIS-1320537

• Supported by gifts from Adobe, Microsoft, Google, and Nokia

88

http://www.cc.gatech.edu/gvu/ii

89