Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K...
Transcript of Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K...
![Page 1: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/1.jpg)
1
Design Study Methodology
Cmpt 767 - Visualization
Steven Bergner
[based on slides by Torsney-Weir / Möller / Munzner]
![Page 2: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/2.jpg)
Acknowledgements
• Tamara’s vis course slides
• Design study methodology: Reflections from the trenches and the stacks. Michael Sedlmair, Mariah Meyer, and Tamara Munzner. IEEE Trans. Visualization and Computer Graphics 18(12):2431-2440, 2012.
• Cluster and Calendar based Visualization of Time Series Data. Jarke J. van Wijk and Edward R. van Selow. Proc. InfoVis 1999, p 4-9.
2
![Page 3: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/3.jpg)
Nested model for vis design:Four Levels for Validation
3[Munzner]
![Page 4: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/4.jpg)
Nested model for vis design
4
What are people doing? What are their goals?
[Munzner]
![Page 5: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/5.jpg)
Nested model for vis design
5
What are data/tasks to accomplish these goals?
[Munzner]
![Page 6: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/6.jpg)
Nested model for vis design
6
How do I show /interact with the data?
[Munzner]
![Page 7: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/7.jpg)
Nested model for vis design
7
How do I make this all work?
[Munzner]
![Page 8: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/8.jpg)
Nested model for vis design: Threats
8[Munzner]
![Page 9: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/9.jpg)
Nested model for vis design: Validation steps
9[Munzner]
![Page 10: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/10.jpg)
Nested model for vis design
10[Munzner]
![Page 11: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/11.jpg)
1111
Workflow to design a tool
![Page 12: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/12.jpg)
Make the right tool
12
van Wijk:1999
Questions
Data
Tasks
http://halalfocus.net/wp-content/uploads/2014/05/question-mark-nothing.jpg
Vis researcher
![Page 13: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/13.jpg)
Make the right tool
13
van Wijk:1999
Questions
Data
TasksDesign study methodology
![Page 14: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/14.jpg)
Design study methodology
14
Sedlmair:2012
![Page 15: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/15.jpg)
Design study definition
15
Design study papers explore the choices made when applying infovis techniques in an application area, for example relating the visual encodings and interaction techniques to the requirements of the target task. Although a limited amount of application domain background information can be useful to provide a framing context in which to discuss the specifics of the target task, the primary focus of the case study must be the infovis content. Describing new techniques and algorithms developed to solve the target problem will strengthen a design study paper, but the requirements for novelty are less stringent than in a Technique paper.
[InfoVis03 CFP, infovis.org/infovis2003/CFP]Munzner
![Page 16: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/16.jpg)
Design study definition
16
Design study papers explore the choices made when applying infovis techniques in an application area, for example relating the visual encodings and interaction techniques to the requirements of the target task. Although a limited amount of application domain background information can be useful to provide a framing context in which to discuss the specifics of the target task, the primary focus of the case study must be the infovis content. Describing new techniques and algorithms developed to solve the target problem will strengthen a design study paper, but the requirements for novelty are less stringent than in a Technique paper.
[InfoVis03 CFP, infovis.org/infovis2003/CFP]Munzner
![Page 17: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/17.jpg)
Design study definition
17
Design study papers explore the choices made when applying infovis techniques in an application area, for example relating the visual encodings and interaction techniques to the requirements of the target task. Although a limited amount of application domain background information can be useful to provide a framing context in which to discuss the specifics of the target task, the primary focus of the case study must be the infovis content. Describing new techniques and algorithms developed to solve the target problem will strengthen a design study paper, but the requirements for novelty are less stringent than in a Technique paper.
[InfoVis03 CFP, infovis.org/infovis2003/CFP]Munzner
![Page 18: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/18.jpg)
Design study definition
18
Design study papers explore the choices made when applying infovis techniques in an application area, for example relating the visual encodings and interaction techniques to the requirements of the target task. Although a limited amount of application domain background information can be useful to provide a framing context in which to discuss the specifics of the target task, the primary focus of the case study must be the infovis content. Describing new techniques and algorithms developed to solve the target problem will strengthen a design study paper, but the requirements for novelty are less stringent than in a Technique paper.
[InfoVis03 CFP, infovis.org/infovis2003/CFP]Munzner
![Page 19: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/19.jpg)
Design study methodology
19
Sedlmair:2012
![Page 20: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/20.jpg)
Design study methodology
20
• Read vis papers
• Read vis books
• Talk to vis practitioners
• This course!
What tools/techniques are available?
![Page 21: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/21.jpg)
Design study methodology
21
• Do they have interesting problems?
• Do they need novel solutions?
• Is there data?
• Can I work with these people?
Are these good collaborators?
![Page 22: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/22.jpg)
When can you do a design study?
22
Sedlmair:2012
![Page 23: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/23.jpg)
Design study methodology
23
• Do people have time for a new project?
• “Front-line analyst” is the domain expert
• Are there false “front-line analysts”?
• Do you need a “translator”?
Who’s who?
![Page 24: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/24.jpg)
Design study methodology
24
• Requirements analysis
• Critical reflection on requirements!
• Abstraction is important for transferability
• Need some domain-expert knowledge
Problem characterization and abstraction
![Page 25: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/25.jpg)
Design study methodology
25
• Overall goal: are there temporal patterns in power consumption?
example: Cluster-Calendar, van Wijk and van Selow
![Page 26: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/26.jpg)
Design study methodology
26
• Data: ~50K pairs of (value, time)• Tasks
• Find standard day patterns• Find out how patterns are distributed over year, week,
season• Find outliers from standard daily patterns• Want overview first, details on demand
example: Cluster-Calendar, van Wijk and van Selow
![Page 27: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/27.jpg)
Design study methodology
27
• Limitations of previous work:
• predictive mathematical models: details lost
• scale-space approaches (wavelet, fourier, fractal): hard to interpret, known scales lost
• 3D mountain diagram (x: hours, y: value, z: days)
example: Cluster-Calendar, van Wijk and van Selow
![Page 28: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/28.jpg)
Design study methodology
28van Wijk:1999
Power demand by ECN, displayed as a function of hours and days
![Page 29: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/29.jpg)
Design study methodology
29
• Pretty, not so useful• Daily, weekly patterns are hard to see
van Wijk:1999
![Page 30: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/30.jpg)
Design study methodology
30
• What data transformations are needed?
• What visual designs to use?
• How to tie this together with interaction?
• Don’t code!
Data abstraction, visual encoding, interaction
![Page 31: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/31.jpg)
Design study methodology
31
• Data transform: hierarchical clustering
example: Cluster-Calendar, van Wijk and van Selow
![Page 32: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/32.jpg)
Design study methodology
32
• Data transform: hierarchical clustering• start with M day patterns
• compute pair-wise differences, merge most similar• now we have M-1 patterns• repeat until we have 1 root cluster
• result: binary hierarchy of clusters
example: Cluster-Calendar, van Wijk and van Selow
![Page 33: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/33.jpg)
Design study methodology
33
• Data transform: hierarchical clustering
• issues:
• distance metric to use?
• how to display the cluster?
example: Cluster-Calendar, van Wijk and van Selow
![Page 34: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/34.jpg)
Design study methodology
34
• dendrogram
example: Cluster-Calendar, van Wijk and van Selow
van Wijk:1999
![Page 35: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/35.jpg)
Design study methodology
35
• dendrogram
example: Cluster-Calendar, van Wijk and van Selow
van Wijk:1999
Shows hierarchical structure but not time distribution!
![Page 36: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/36.jpg)
36Overview Detail van Wijk:1999
Design study methodology
![Page 37: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/37.jpg)
Design study methodology
37
• clusters: data transformation to aggregate data
• calendar: familiar visual representation for time
• linking: interactive exploration of the data
• task analysis guided choices: 3D extrusion and dendrogram don’t work
example: Cluster-Calendar, van Wijk and van Selow
![Page 38: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/38.jpg)
Design study methodology
38
• Need to test design hypotheses
• Rapid prototyping (will probably throw away alot of code)
• Breaking bugs vs annoying bugs
• Fast usability testing
Yay coding!
![Page 39: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/39.jpg)
Design study methodology
39
• Domain experts need to play with software
• What works, what doesn’t?
• How to evaluate?
• May need to redesign/reimplement a lot
Hand-off to the users
![Page 40: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/40.jpg)
Design study methodology
40
Critique?
van Wijk:1999
![Page 41: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/41.jpg)
Design study methodology
Compare to existing design guidelines Confirm which ones worked Reject which ones didn’t work Come up with new guidelines
41
Refine, reject, propose guidelines
![Page 42: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/42.jpg)
Design study methodology
Forces clear articulation of problem, tasks, solution Who else does my study help? - transferability! Think carefully about what readers will care about This takes time to do well!
42
Yay words!
![Page 43: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/43.jpg)
Make the right tool
43
van Wijk:1999
Questions
Data
TasksDesign study methodology
![Page 44: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/44.jpg)
Where are design studies?
44[Munzner]
![Page 45: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/45.jpg)
[Munzner]
Where are design studies?
45
![Page 46: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/46.jpg)
46
Pitfalls
![Page 47: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/47.jpg)
Pitfalls
47
#1: Don’t skip steps!
![Page 48: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/48.jpg)
Pitfalls
48
• insufficient knowledge of literature
![Page 49: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/49.jpg)
Pitfalls
49
• collaboration with the wrong people
• no real data available
• insufficient time available from collaborators
• no need for visualization: automate
• no need for research: engineering project
![Page 50: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/50.jpg)
Pitfalls
50
• is this interesting to me?
• existing tools are good enough
• not an important/recurring task
• no rapport with collaborators
![Page 51: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/51.jpg)
Pitfalls
51
• not identifying front-line analyst and gatekeeper
• assuming same role distribution across projects
• mistaking tool-builders for real end users
![Page 52: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/52.jpg)
Pitfalls
52
• ignoring practices that currently work well
• expecting just talking or fly on the wall to work
• domain experts design the visualizations
• too much/too little domain knowledge
![Page 53: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/53.jpg)
Pitfalls
53
• too little abstraction
• design consideration space too small
• mistaking technique-driven and problem-driven work
![Page 54: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/54.jpg)
Pitfalls
54
• non-rapid prototyping
• usability: too little/too much
![Page 55: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/55.jpg)
Pitfalls
55
• insufficient deploy time
• non-real task/data/user
• liking a tool is not validation!
![Page 56: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/56.jpg)
Pitfalls
56
• failing to improve guidelines
![Page 57: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/57.jpg)
Pitfalls
57
• not enough writing time
• no technique contribution ≠ write a design study
• too much domain background
• chronological story vs concentrating on results
• premature end to the project
![Page 58: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/58.jpg)
Additional reading
58
• Design study methodology: Reflections from the trenches and the stacks. Michael Sedlmair, Mariah Meyer, and Tamara Munzner. IEEE Trans. Visualization and Computer Graphics 18(12):2431-2440, 2012.
• Cluster and Calendar based Visualization of Time Series Data. Jarke J. van Wijk and Edward R. van Selow. Proc. InfoVis 1999, p 4-9.
![Page 59: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/59.jpg)
59
Evaluating Information Visualisations
![Page 60: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/60.jpg)
Sources
Evaluating Information Visualizations. Sheelagh Carpendale. Chapter in Information Visualization: Human-Centered Issues and Perspectives, Springer LNCS 4950, 2008, p 19-45.
Tamara Munzner’s Course Slides on Evaluation
60
![Page 61: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/61.jpg)
Psychophysics
method of limitso find limitations of human perceptions
error detection methodso find threshold of performance degradationo staircase procedure to nd threshold faster
method of adjustmento find optimal level of stimuli by letting subjects control
the level
61
![Page 62: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/62.jpg)
Cognitive Psychology
repeating simple, but important tasks, and measure reaction time or erroro Miller’s 7+/- 2 short-term memory experimentso Fitts’ Law (target selection)o Hick’s Law (decision making given n choices)
interference between channels multi-modal studies
o MacLean 2005, Perceiving Ordinal Data HapticallyUnder Workload
using haptic feedback for interruption when the participants were visually (and cognitively) busy
62
![Page 63: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/63.jpg)
Structural Analysis
requirement analysis, task analysis structured interviews
o can be used almost anywhere, for open-ended questions and answers
rating/Likert scaleso commonly used to solicit subjective feedbacko ex: NASA-TLX (Task Load Index) to assess mental
workload• “it is frustrating to use the interface”
Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree
63
![Page 64: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/64.jpg)
Comparative User Studies
study design: factors and levels factors
o independent variableso ex: interface, task, participant demographics
levelso number of values in each factoro limited by length of study and number of participants
64
![Page 65: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/65.jpg)
Comparative User Studies
study design: within, or between? within
o everybody does all the conditionso can lead to ordering effectso can account for individual differences and reduce noiseo thus can be more powerful and require fewer participantso combinatorial explosion
• severe limits on number of conditionso possible workaround is multiple sessions
betweeno divide participants into groups
each group does only some conditions
65
![Page 66: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/66.jpg)
Comparative User Studies
measurements (dependent variables)o performance indicators: task completion time, error rates,
mouse movemento subjective participant feedback: satisfaction ratings, closed-
ended questions, interviewo observations: behaviors, signs of frustration
number of participantso depends on effect size and study design: power of
experiment possible confounds?
o learning effect: did everybody use interfaces in a certain order?
if so, are people faster because they are more practiced, or because of true interface effect?
66
![Page 67: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/67.jpg)
Comparative User Studies
result analysiso should know how to analyze the main
results/hypotheses BEFORE studyo hypothesis testing analysis (using ANOVA or t-tests)
tests how likely observed differences between groups are due to chance alone
o ex: a p-value of 0.05 means there is a 5% probability the difference occurred by chance
usually good enough for HCI studies pilots!
o should have good idea of forthcoming results of the study BEFORE running actual study trials
67
![Page 68: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/68.jpg)
Evaluation Throughout Design Cycle
user/task centered design cycleo initial assessmentso iterative design processo benchmarkingo deployment
identify problems, go back to previous step
Task-Centered User Interface Design, Clayton Lewis and John Rieman, Chapters 0-5.
68
![Page 69: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/69.jpg)
Initial Assessments
what kind of problems are the system aiming to address?o analyze a large and complex dataset
who are your target users?o data analysts
what are the tasks? what are the goals?o find trends and patterns in the data via exploratory analysis
what are their current practiceso statistical analysis
why and how can visualization be useful?o visual spotting of trends and patterns
talk to the users, and observe what they do task analysis
69
![Page 70: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/70.jpg)
Iterative Design Process
does your design address the users’ needs? can they use it? where are the usability problems? evaluate without users
o cognitive walkthrougho action analysiso heuristics analysis
evaluate with userso usability evaluations (think-aloud)
bottom-line measurements
70
![Page 71: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/71.jpg)
Benchmarking
how does your system compare to existing ones? empirical, comparative studies
o ask specific questionso compare an aspect of the system with specific tasks
• Amar/Stasko task taxonomy papero quantitative, but limited
• The Challenge of Information Visualization Evaluation, Catherine Plaisant, Proc. AVI 2004
71
![Page 72: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/72.jpg)
Deployment
how is the system used in the wild? how are people using it? does the system fit into existing work flow? environment?
contextual studies, field studies
72
![Page 73: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/73.jpg)
Compare Systems vs. Characterize Usage
user/task centered design cycle:o initial assessmentso iterative design processo benchmarking: head-to-head comparisono deploymento (identify problems, go back to previous step)
understanding/characterizing techniqueso tease apart factorso when and how is technique appropriate
line is blurry: intent
73
![Page 74: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/74.jpg)
Perceptual Scalability
what are perceptual/cognitive limits when screen-space constraints lifted?o 2 vs. 32 M pixel displayo macro/micro views
perceptually scalableo no increase in task completion times when normalize to
amount of data
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]74
![Page 75: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/75.jpg)
Embedded Visualizations
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]75
![Page 76: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/76.jpg)
Small Multiples Visualization
Attribute-centric instead of space-centric
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]76
![Page 77: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/77.jpg)
Perceptual Scalability
designo 2 display sizes, between-subjects
• (data size also increased proportionally)o 3 visualization designs, within
• small multiples: bars• embedded graphs• embedded bars
o 7 tasks, withino 42 tasks per participant
• 3 vis x 7 tasks x 2 trials
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]77
![Page 78: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/78.jpg)
Results
20x increase in data, but only 3x increase in absolute task times
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]78
![Page 79: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/79.jpg)
Results
20x increase in data, but only 3x increase in absolute task times
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]79
![Page 80: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/80.jpg)
Results
visual encoding important on small displayso DS: mults sig slower than graphs on smallo DS: mults sig slower than embedded on largeo OS: bars sig faster than graphs for smallo OS: no sig dierence bars/graphs for large
spatial grouping important on large displayso embedded sig faster+preferred over small multo no bar/graph differences
[The Perceptual Scalability of Visualization. Beth Yost and Chris North. IEEE TVCG 12(5) (Proc. InfoVis 06), Sep 2006, p 837-844.]80
![Page 81: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/81.jpg)
Trends: Animation, Trails, Small Multiples
Gapminder: animated bubble charts + humano x/y position, size, color,
animationo is animation effective?
• presentation vs analysis• trend vs transitions
[Effectiveness of Animation in Trend Visualization. Robertson et al. IEEE TVCG (Proc. InfoVis 2008). 14(6): 1325-1332 (2008)]81
![Page 82: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/82.jpg)
Trends
many countertrends lost in clutter
[Effectiveness of Animation in Trend Visualization. Robertson et al. IEEE TVCG (Proc. InfoVis 2008). 14(6): 1325-1332 (2008)]82
![Page 83: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/83.jpg)
Small Multiples
individual plots get small
[Effectiveness of Animation in Trend Visualization. Robertson et al. IEEE TVCG (Proc. InfoVis 2008). 14(6): 1325-1332 (2008)]83
![Page 84: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/84.jpg)
Design
2 use: presentation vs. analysis (between-subjects)o 3 vis encodings: animation vs. traces vs. small mults
2 dataset size: small vs. largeo 3 encoding x 2 size: within-subjects
24 tasks per participanto 4 tasks x 3 encodings x 2 sizes
[Effectiveness of Animation in Trend Visualization. Robertson et al. IEEE TVCG (Proc. InfoVis 2008). 14(6): 1325-1332 (2008)]84
![Page 85: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/85.jpg)
Results
small multiples more accurate than animation animation faster for presentation, slower for analysis than
small multiples and trends dataset size matters (unsurprisingly)
[Effectiveness of Animation in Trend Visualization. Robertson et al. IEEE TVCG (Proc. InfoVis 2008). 14(6): 1325-1332 (2008)]85
![Page 86: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/86.jpg)
User Study Goals
compare systems characterize methods formative feedback summative judgement convince stakeholders
86
![Page 87: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/87.jpg)
87
Thank you for your attention!
![Page 88: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/88.jpg)
88
Notes fromS. Carpendale’s Evaluating Information Visualizations
Supplementary Slides
![Page 89: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/89.jpg)
Notes from the paper
current evaluations not convincing enough to encourage widespread adoption of information visualization toolso Small datasetso University participants onlyo Simple tasks
Rather: real users & tasks, complex datasets
![Page 90: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/90.jpg)
Challenges of Evaluations
Difficult to pick right focus and ask the right questions Choose right methodology, sufficiently rigorous in procedure and data collection Info vis research relates to other empirical research
o Human computer interaction (HCI)• Many tasks are interface interaction• Usability aspects: access to visual representation and underlying dataset• Appropriate sample of participants (domain experts)• Results due to particular technique or overall solution?• Participants familiarity with existing approaches
o Perceptual psychology• Appropriate representational encoding, readability of visuals
o Cognitive reasoning - Tasks vary with data type• Low- level: compare, associate, rank, cluster, correlate, categorize• High-level: understand data trends, uncertainties, causal relationships,
predicting the future, or learning a domain
![Page 91: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/91.jpg)
Challenge: Does vis promote insight into the data?
Discover the unexpectedo Often long-term, on-goingo “answering questions you didn’t know you had”o Depends on participant – motivation, knowledge,
experience
![Page 92: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/92.jpg)
Choose an Evaluation Approach
Generalizability
o Results apply toother people?
Precision
o Definite measurements
o Controlled factors
Realism
[Carpendale ‘08, adapted from McGrath Methodology Matters]
![Page 93: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/93.jpg)
Approaches glossary
Field studyo conducted in actual situation, unobtrusive observer
Field experimento ask participants to perform specific tasko more precision, less realism
Lab experimento may add ‘thinkaloud’ protocol
Experimental Simulationo reintroduce some realism via sim, avoid risky or unethical situations
Judgement studyo person’s response to a set of stimuli under ‘neutral conditions’o e.g. what surface texture (or shading) is better to interpret shape?
Sample surveyo discover relationships between variables in a given population
Formal theoryo possibly meta-study to further infovis theory
Computer simulationo May not involve participants
![Page 94: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/94.jpg)
Quantitative Methodology
Hypothesis developmento Precise questions of broad interest
Identification of independent varso Factors that may affect hypothesis, ideally few
Control of independent varso How indep. vars are changed, experimental design
Elimination of complexityo Controlled environment
Measurement of dependent varso Common metrics: speed, accuracy, error rate, satisfaction
Statistical analysis
![Page 95: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/95.jpg)
Qualitative Methods
Observation techniqueso Take notes unobtrusively, maybe during breako Note overt and covert in activities and communicationso Use only one side of the note-paper
Interview techniqueso Make sure you understando Limit your inclination to talko Listen for and encourage the less formal, less guarded
expression of participant’s thoughtso Avoid leading Q’s, ask open Q’s, ask for concrete detailso Humanity of interview, be present, aware, sensitive
![Page 96: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/96.jpg)
Qualitative Methodologies
Nested qualitative methods, part of quantitative studyo Log Experimenter Observations during qualitative studyo Think-aloud protocol (may be unnatural to participants)o Collect participants’ opinions
![Page 97: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/97.jpg)
Qualitative methodologies (cont’d)
Inspection Evaluation Methods conducted by expertso Usability heuristics, e.g.
• visibility of system status• match between system and real world• personal control and freedom• consistency and standards• error prevention and handling, help and documentation• recognition rather than recall• aesthetic and minimalist design
o Collaboration heuristics• communication and coordination• distributed or co-located
![Page 98: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/98.jpg)
Primarily qualitative study
In situ observational studyo “fly on the wall”
Participatory observationo Collaborate with domain experts
Laboratory observational study Contextual interview
o Ask about task, setting, or application of interest
Lessen the task and data comprehension divide between visualization experts and the domain experts
![Page 99: Design study methodology - stevenbergner.github.io...Design study methodology. 26 • Data: ~50K pairs of (value, time) • Tasks • Find standard day patterns • Find out how patterns](https://reader033.fdocuments.us/reader033/viewer/2022050104/5f42cecbe37d30537a0c2467/html5/thumbnails/99.jpg)
Challenges of qualitative study
Sample size Subjectivity Analysing qualitative data
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted” – Albert Einstein