Thinking with Visualizations: Cognitive Execution of Visual Queries
description
Transcript of Thinking with Visualizations: Cognitive Execution of Visual Queries
![Page 1: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/1.jpg)
Thinking with Visualizations:Cognitive Execution of Visual Queries
Colin WareData Visualization Research Lab
University of New Hampshire
Designing with cyborgs in mind
![Page 2: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/2.jpg)
![Page 3: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/3.jpg)
Architecture for visual thinking
DisplayFeatures
Proto-objects andPatterns
VisualWorkingMemory
GIST
VisualQuery
VerbalWorkingMemory
Egocentric object andPattern map
Interaction Loop
10+ billion neuronsParallel, automatic
![Page 4: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/4.jpg)
![Page 5: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/5.jpg)
Stage 2 Pattern perception
DisplayFeatures
Proto-objects andPatterns
VisualWorkingMemory
GIST
VisualQuery
VerbalWorkingMemory
Egocentric object andPattern map
Interaction Loop
10 billion neuronsParallel, automatic
![Page 6: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/6.jpg)
Stage 2 Pattern perception
Visual queries are executed by finding patterns in displays
Gestalt principles Proximity Continuity Connectedness Closure
![Page 7: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/7.jpg)
Basic patterns
OutlierOverlap
Groups
Connection or Relationship
Identity
Uptrend
Size = quantity
Contrast = quantity
![Page 8: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/8.jpg)
Dual Coding TheoryVerbal-Propositional Information Visual Structural
Information
VerbalWorking Memory
BaddeleyVisual
Working Memory
Central Executive
Visuo-SpatialSketchpad
Verbal input Visual input
Words and symbols
![Page 9: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/9.jpg)
Pictures and Words
When should we use a visual display? What is a visual language?
![Page 10: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/10.jpg)
Visual and verbal pseudo-code
While letters in stack Take a letter Put a stamp on it Put it in the ‘out tray’
While (text in input){
getline;change characters to upper case;write to output;
}
Visual programming languages have a history of failure
get line of textfrom input file
change charactersto upper case
write line to outputfile
more input?yes
no
![Page 11: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/11.jpg)
Capacity of visual working memory (Vogal, Woodman, Luck, 2001)
Task – change detection Can see 3.3 objects Each object can be complex
1 second
![Page 12: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/12.jpg)
Sequential comparison task
![Page 13: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/13.jpg)
![Page 14: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/14.jpg)
![Page 15: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/15.jpg)
![Page 16: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/16.jpg)
![Page 17: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/17.jpg)
Change Blindness
Simons and Levin
![Page 18: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/18.jpg)
Architecture for visual thinking
DisplayFeatures
Proto-objects andPatterns
VisualWorkingMemory
GIST
VisualQuery
VerbalWorkingMemory
Egocentric object andPattern map
Interaction Loop
10 billion neuronsParallel, automatic
Object Files
SelectiveTuning
Tsotsos
![Page 19: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/19.jpg)
Object File
Egocentric Coordinate Map
Object File
Object File
Gist Semantic content
![Page 20: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/20.jpg)
Visual search
a
VisualSearch orMonitoringStrategy
EyeMovementControl
Useful VisualField of View
![Page 21: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/21.jpg)
Eye movements
Two or three a second Preserves Context
The screen is a kind of buffer for visual ideas – we cannot see it all at once but we can sample it rapidly
![Page 22: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/22.jpg)
Thinking visuallyEmbedded processes
Define problem and steps to solution Formulate parts of problem as visual
questions/hypotheses Setup search for patterns
Eye movement control loop IntraSaccadic Scanning Loop
(form objects)
![Page 23: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/23.jpg)
![Page 24: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/24.jpg)
Problem
Trip Port Bou- Calais (5 days 3 cities) Visual Problem Mayor Highways
– Distance < 1.2 min = red smooth path Eye movements to identify major candidate pathways
Pattern Identification: smooth, red, connected segments / reject non-red-wrong direction
Part solutions into vwm – spatial markers Parts may be handed to verbal wm
![Page 25: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/25.jpg)
Software Engineering Example - with Graph Representation
Segment Big Module into parts High Cohesion (semantics) Low Coupling
Find highly connected subgraphs with minimal links Scan for candidate patterns
Look for Low connectivity Look for Semantic similarity (symbols)
Important question: what are relevant pattern that can fit in vwm
![Page 26: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/26.jpg)
Cost of Knowledge
Intra-saccade (0.04 sec) (Query execution) An eye movement (0.5 sec) < 10 deg : 1 sec>
20 deg. A hypertext click (1.5 sec but loss of context) A pan or scroll (3 sec but we don’t get far) Walking (30 sec. we don’t get far) Flying (faster can be tuned) Zooming, fisheye, DragMag
![Page 27: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/27.jpg)
Walking Flying (30 sec +)Naïve view that does not take perception or the
cost of action into account.
Semnet GraphVisualizer3D ConeTrees
![Page 28: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/28.jpg)
How to navigate large 2 ½D spaces? (Matt Plumlee) Zooming Vs Multiple Windows
Key problem: How can we keep focus and maintain context.
Focus is what we are attending to now. Context is what we may wish to attend to.
2 solutions: Zooming, multiple windows
![Page 29: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/29.jpg)
When is zooming better thanmultiple windows
Key insight: Visual working memory is a very limited resource. Only 3 objects
GeoZui3D
![Page 30: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/30.jpg)
Task: searching for target patterns that match
![Page 31: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/31.jpg)
Cognitive Model (grossly simplified)
Time = setup cost + number of visits*time per visit
Number of visits is a function of number of objects (& visual complexity)
When there are too many multiple visits are needed
![Page 32: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/32.jpg)
Prediction Results
As targets (and visual working memory load) increases, multiple
Windows become more attractive.
![Page 33: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/33.jpg)
Lessons for design
Tells us when we need extra windows What do we need to keep on the screen at
the same time Simultaneous vs successive views
![Page 34: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/34.jpg)
Need low cost and low cognitive cost interactions
Constellation: Hover queries (Munzner)
MEGraph
BrushingDynamic Queries
![Page 35: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/35.jpg)
Architecture for visual thinking
DisplayFeatures
Proto-objects andPatterns
VisualWorkingMemory
GIST
VisualQuery
VerbalWorkingMemory
Egocentric object andPattern map
Interaction Loop
Low -Medium level – pattern perception
High level vwm and cognitive costs
Appropriate representation (words vs images)
Navigation costs
![Page 36: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/36.jpg)
Cognitive Systems
Slogan: “Tighten the loop” Take a systems approach Visualization for pattern finding Take into account the non-homogeneity of space. Algorithms for pre-filtering Optimize navigation – brushing, dynamic queries –
breadth first range searches
Large high resolution displays
![Page 37: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/37.jpg)
Acknowledgements
NSERC (Canada) NSF (USA) NOAA ARDA
![Page 38: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/38.jpg)
![Page 39: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/39.jpg)
3D versus 2D
34% memoryerrors
20% memoryerrors
21% errors 5.1 secsub-structure
11.4% errors 3.7 secsub-structure
![Page 40: Thinking with Visualizations: Cognitive Execution of Visual Queries](https://reader036.fdocuments.us/reader036/viewer/2022062722/56813ac3550346895da2d3bc/html5/thumbnails/40.jpg)
Research topicWhat are easy visual queries
Easy= single object comparison in vwm