A Nested Model for Visualization Design and Validation · [test on any users, informal usability...
Transcript of A Nested Model for Visualization Design and Validation · [test on any users, informal usability...
![Page 1: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/1.jpg)
A Nested Model for VisualizationDesign and Validation
Tamara MunznerUniversity of British ColumbiaDepartment of Computer Science
![Page 2: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/2.jpg)
2
How do you show your system is good?
• so many possible ways!• algorithm complexity analysis• field study with target user population• implementation performance (speed, memory)• informal usability study• laboratory user study• qualitative discussion of result pictures• quantitative metrics• requirements justification from task analysis• user anecdotes (insights found)• user community size (adoption)• visual encoding justification from theoretical principles
![Page 3: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/3.jpg)
3
Contribution
• nested model unifying design and validation
• guidance on when to use what validation method
• different threats to validity at each level of model
• recommendations based on model
![Page 4: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/4.jpg)
4
Four kinds of threats to validity
![Page 5: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/5.jpg)
5
Four kinds of threats to validity
domain problem characterization
• wrong problem• they don’t do that
![Page 6: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/6.jpg)
6
Four kinds of threats to validity
domain problem characterization data/operation abstraction design
• wrong problem• they don’t do that
• wrong abstraction• you’re showing them the wrong thing
![Page 7: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/7.jpg)
7
Four kinds of threats to validity
domain problem characterization data/operation abstraction design encoding/interaction technique design
• wrong problem• they don’t do that
• wrong abstraction• you’re showing them the wrong thing
• wrong encoding/interaction technique• the way you show it doesn’t work
![Page 8: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/8.jpg)
8
Four kinds of threats to validity
domain problem characterization data/operation abstraction design encoding/interaction technique design algorithm design
• wrong problem• they don’t do that
• wrong abstraction• you’re showing them the wrong thing
• wrong encoding/interaction technique• the way you show it doesn’t work
• wrong algorithm• your code is too slow
![Page 9: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/9.jpg)
9
threat: wrong problem
threat: bad data/operation abstraction threat: ineffective encoding/interaction technique
threat: slow algorithm
• each validation works for only one kind of threat to validity
Match validation method to contributions
![Page 10: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/10.jpg)
10
Analysis examples
justify encoding/interaction design
qualitative result image analysistest on target users, get utility anecdotes
justify encoding/interaction design
measure system time/memoryqualitative result image analysis
computational complexity analysis
qualitative/quantitative image analysis
lab study, measure time/errors for operation
Interactive visualization of genealogical graphs.McGuffin and Balakrishnan. InfoVis 2005.
MatrixExplorer. Henry and Fekete. InfoVis 2006.
An energy model for visual graph clustering. (LinLog)Noack. Graph Drawing 2003
Flow map layout. Phan et al. InfoVis 2005.
LiveRAC. McLachlan, Munzner, Koutsofios,and North. CHI 2008.
Effectiveness of animation in trend visualization.Robertson et al. InfoVis 2008.
measure system time/memoryqualitative result image analysis
observe and interview target users
justify encoding/interaction design
qualitative result image analysis
observe and interview target users
justify encoding/interaction design
field study, document deployed usage
![Page 11: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/11.jpg)
11
Nested levels in model
domain problem characterization data/operation abstraction design encoding/interaction technique design algorithm design
• output of upstream levelinput to downstream level
• challenge: upstream errors inevitably cascade• if poor abstraction choice made, even perfect technique
and algorithm design will not solve intended problem
![Page 12: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/12.jpg)
12
Characterizing domain problems
• tasks, data, workflow of target users• problems: tasks described in domain terms• requirements elicitation is notoriously hard
problem data/op abstraction enc/interact technique algorithm
![Page 13: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/13.jpg)
13
Designing data/operation abstraction
• mapping from domain vocabulary/concerns to abstraction• may require transformation!
• data types: data described in abstract terms• numeric tables, relational/network, spatial, ...
• operations: tasks described in abstract terms• generic
• sorting, filtering, correlating, finding trends/outliers...• datatype-specific
• path following through network...
problem data/op abstraction enc/interact technique algorithm
![Page 14: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/14.jpg)
14
Designing encoding,interaction techniques
• visual encoding• marks, attributes, ...• extensive foundational work exists
• interaction• selecting, navigating, ordering, ...• significant guidance exists
Semiology of Graphics. Jacques Bertin, Gauthier-Villars 1967, EHESS 1998
problem data/op abstraction enc/interact technique algorithm
![Page 15: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/15.jpg)
15
Designing algorithms
• well-studied computer science problem• create efficient algorithm given clear specification• no human-in-loop questions
problem data/op abstraction enc/interact technique algorithm
![Page 16: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/16.jpg)
16
Immediate vs. downstream validation
threat: wrong problem
threat: bad data/operation abstraction threat: ineffective encoding/interaction technique
threat: slow algorithm
implement system
![Page 17: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/17.jpg)
17
Domain problem validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique
threat: slow algorithm
implement system
• immediate: ethnographic interviews/observations
![Page 18: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/18.jpg)
18
Domain problem validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique
threat: slow algorithm
implement system
validate: observe adoption rates
• downstream: adoption (weak but interesting signal)
![Page 19: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/19.jpg)
19
Abstraction validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique
threat: slow algorithm
implement system
validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• downstream: can only test with target users doing real work
![Page 20: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/20.jpg)
20
Encoding/interaction technique validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
implement system
validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• immediate: justification useful, but not sufficient - tradeoffs
![Page 21: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/21.jpg)
21
Encoding/interaction technique validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
implement system
validate: qualitative/quantitative result image analysis
validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• downstream: discussion of result images very common
![Page 22: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/22.jpg)
22
Encoding/interaction technique validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
implement system
validate: qualitative/quantitative result image analysis
validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• downstream: studies add another level of rigor (and time)
![Page 23: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/23.jpg)
23
Encoding/interaction technique validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
implement system
validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• usability testing necessary for validity of downstream testing• not validation method itself!
![Page 24: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/24.jpg)
24
Algorithm validation
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
validate: analyze computational complexity implement system validate: measure system time/memory validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
• immediate vs. downstream here clearly understood in CS
![Page 25: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/25.jpg)
25
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
validate: analyze computational complexity implement system validate: measure system time/memory validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
Avoid mismatches• can’t validate encoding with wallclock timings
![Page 26: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/26.jpg)
26
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
validate: analyze computational complexity implement system validate: measure system time/memory validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
Avoid mismatches• can’t validate abstraction with lab study
![Page 27: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/27.jpg)
27
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
validate: analyze computational complexity implement system validate: measure system time/memory validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
Single paper would include only subset• can’t do all for same project
• not enough space in paper or time to do work
![Page 28: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/28.jpg)
28
threat: wrong problem validate: observe and interview target users threat: bad data/operation abstraction threat: ineffective encoding/interaction technique validate: justify encoding/interaction design threat: slow algorithm
validate: analyze computational complexity implement system validate: measure system time/memory validate: qualitative/quantitative result image analysis [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test on target users, collect anecdotal evidence of utility validate: field study, document human usage of deployed system validate: observe adoption rates
Single paper would include only subset• pick validation method according to contribution claims
![Page 29: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/29.jpg)
29
Real design process
• iterative refinement• levels don’t need to be done in strict order• intellectual value of level separation
• exposition, analysis
• shortcut across inner levels + implementation• rapid prototyping, etc.
• low-fidelity stand-ins so downstream validation canhappen sooner
![Page 30: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/30.jpg)
30
Related work
• influenced by many previous pipelines• but none were tied to validation
• [Card, Mackinlay, Shneiderman 99], ...
• many previous papers on how to evaluate• but not when to use what validation methods
• [Carpendale 08], [Plaisant 04], [Tory and Möller 04]
• exceptions• good first step, but no formal framework
[Kosara, Healey, Interrante, Laidlaw, Ware 03]• guidance for long term case studies, but not other contexts
[Shneiderman and Plaisant 06]• only three levels, does not include algorithm
[Ellis and Dix 06], [Andrews 08]
![Page 31: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/31.jpg)
31
Recommendations: authors
• explicitly state level of contribution claim(s)
• explicitly state assumptions for levels upstream ofpaper focus• just one sentence + citation may suffice
• goal: literature with clearer interlock between papers• better unify problem-driven and technique-driven work
![Page 32: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/32.jpg)
32
Recommendation: publication venues
• we need more problem characterization• ethnography, requirements analysis
• as part of paper, and as full paper• now full papers relegated to CHI/CSCW
• does not allow focus on central vis concerns
• legitimize ethnographic “orange-box” papers!observe and interview target users
![Page 33: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/33.jpg)
33
Lab study as core now deemed legitimate
justify encoding/interaction design
qualitative result image analysistest on target users, get utility anecdotes
justify encoding/interaction design
measure system time/memoryqualitative result image analysis
computational complexity analysis
qualitative/quantitative image analysis
lab study, measure time/errors for operation
Interactive visualization of genealogical graphs.McGuffin and Balakrishnan. InfoVis 2005.
MatrixExplorer. Henry and Fekete. InfoVis 2006.
An energy model for visual graph clustering. (LinLog)Noack. Graph Drawing 2003
Flow map layout. Phan et al. InfoVis 2005.
LiveRAC. McLachlan, Munzner, Koutsofios,and North. CHI 2008.
Effectiveness of animation in trend visualization.Robertson et al. InfoVis 2008.
measure system time/memoryqualitative result image analysis
observe and interview target users
justify encoding/interaction design
qualitative result image analysis
observe and interview target users
justify encoding/interaction design
field study, document deployed usage
![Page 34: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/34.jpg)
34
Limitations
• oversimplification
• not all forms of user studies addressed
• infovis-oriented worldview
• are these levels the right division?
![Page 35: A Nested Model for Visualization Design and Validation · [test on any users, informal usability study] validate: lab study, measure human time/errors for operation validate: test](https://reader036.fdocuments.us/reader036/viewer/2022071023/5fd8230ae9b1e966b6396091/html5/thumbnails/35.jpg)
35
Conclusion
• new model unifying design and validation• guidance on when to use what validation method• broad scope of validation, including algorithms
• recommendations• be explicit about levels addressed and state
upstream assumptions so papers interlock more• we need more problem characterization work
these slides posted at http://www.cs.ubc.ca/~tmm/talks.html#iv09