Pitfalls of Information Access with Visualizations in Remote Collaborative Analysis
-
Upload
aruna-balakrishnan -
Category
Documents
-
view
280 -
download
0
Transcript of Pitfalls of Information Access with Visualizations in Remote Collaborative Analysis
Pitfalls of Information Access with Visualizationsin Remote Collaborative Analysis
Aruna D. Balakrishnan, Susan R. Fussell, Sara Kiesler & Aniket KitturHuman Computer Interaction InstituteCarnegie Mellon UniversityCarnegie Mellon University
February 10, 2010
Synchronous Remote Collaborations
di ib d dH d fi distributed datainformation overload
How do you figure out what’s important?
3
Visualizations
textual & numerical data
visualizations
4
visualizations
Interactive Social Network Diagrams
Larkin & Simon 1987Sparrow1998Viégas, Wattenberg, & Dave 2004
5
g , g,Balakrishnan, Fussell, & Kiesler 2008
Shared Information
BenefitsEliminates need f r data e chan eEliminates need for data exchangeFocus on understanding & analysis of materialsSh d l d l Shared mental model (Fraidin 2004; Blockeel & Moyle 2002; Kozlowski, Ilgen & Klimoski 2006)
DisadvantagesC fi i bi Confirmation bias (Mojzisch & Schulz-Hardt 2005)
6
Distributed Information
BenefitsIncreased disc ssi n d e t f rced inf e chan eIncreased discussion due to forced info exchangeMultiple perspectives
DisadvantagesForced information exchange
7
Hypotheses
Hypothesis 1Pairs of analysts will perform better when each partner has partial Pairs of analysts will perform better when each partner has partial
data than when each partner has all the data.
Hypothesis 2 Pairs of analysts with a visualization tool will outperform analysts
without a visualization tool.
H h i 3Hypothesis 3Visualizations will benefit collaborative analysis more when each
partner has partial data than when each partner has all the datapartner has partial data than when each partner has all the data.
8
Experiment Design
Detective mystery experimental paradigm Pairs of participants investigated multiple crimes (Scupelli et. al, 2005)p p g p ( p , )
Shared task To find a serial killer responsible for several of these crimesTo find a serial killer responsible for several of these crimes
Distributed evidence E id f i l kill di t ib t d 7 Evidence for serial killer distributed among 7 casesOnly 5 of 7 cases had relevant information to the task
R ll b iRemote collaborationSeated apart to simulate distributed workAccess to Instant MessengerAccess to Instant Messenger
10
Experiment Task
11
Experiment Design
Between-subjects
2 x 3 factorial designPairs randomly assigned to
1 of 2 information access conditions1 of 3 visualization conditions1 of 3 visualization conditions
12
Experimental ConditionsVisualization Conditions
Shared VisUnshared VisNo VisInfo Access Conditions
Split Data
N = 15 N = 15 N = 15
All All Data N = 15N = 15 N = 15
13
No Visualization
Spreadsheet Case file
IM
15
IM
Unshared Visualization
Visualization Case file
IM
16
IM
Shared Visualization
Case fileVisualization
IM
17
IM
Experiment Details
180 ParticipantsAge range from 18 64 yearsAge range from 18 – 64 yearsFluency in English required
Procedure30 minutes
Informed consentTrained in detective work Trained in visualization useTrained in visualization use
60 minutesSerial Killer task
18
Dependent Measures
S l ti d d T k P f Solutions recorded in final reports
Task PerformanceSolve serial killer task
C mm nicati nInstant messenger
logs
CommunicationAmount of communicationTypes of communication gypes o co u cat o(intercoder reliability kappa = .71)
19
Results: Average number of IM words
500
600 Split DataAll Data
400
500 All Data
Average # IM
200
300# IM words
per
0
100
N U h d Sh d
person
None Unshared Shared
( 0 0 )
21
Visualization Condition(p < 0.01)
Results: Split Data Pair Solve Rate
100%Split Data
Percentof Pairs
80%
60%of Pairs Solving
the Case40%
N U h d Sh d0%
20%
None Unshared Shared
22
Visualization Condition
Results: All Data Pair Solve Rate
100%Split DataAll Data
Percentof Pairs
80%
60%
All Data
of Pairs Solving
the Case40%
N U h d Sh d0%
20%
None Unshared Shared
23
Visualization Condition
Results: Communication
# Hypotheses SharedDiscussion of a new hypothesis regarding the serial killer case
I think these four blunt instrument victims areinstrument victims areconnected.
I feel like it is a suspicious man on the b sbus.
24
Results: Hypotheses Discussion
4.0Split DataAll Data
3.0
hese
s se
d rs
on
All Data
1 0
2.0
Hyp
oth
disc
uspe
r pe
r
0.0
1.0
N U h d Sh dNone Unshared Shared
25
Visualization Condition
Results: Correct and Incorrect Solutions
Solved100%
Split All Split All Split All
Incorrect SuspectIndividual
solution
80%
60%
Suspected pattern
solution rates by category
40%
Did not solve
N U h d Sh d0%
20%
None Unshared Shared
26
Visualization Condition
Results: Correct and Incorrect Solutions
Solved100%
Split All Split All Split All
Incorrect SuspectIndividual
solution
80%
60%
Suspected pattern
solution rates by category
40%
Did not solve
N U h d Sh d0%
20%
None Unshared Shared
27
Visualization Condition
Results: Correct and Incorrect Solutions
Solved100%
Split All Split All Split All
confirmation bias
Incorrect SuspectIndividual
solution
80%
60%
Suspected pattern
solution rates by category
40%
Did not solve
N U h d Sh d0%
20%
None Unshared Shared
28
Visualization Condition
Results: Correct and Incorrect Solutions
Solved100%
Split All Split All Split All
Incorrect SuspectIndividual
solution
80%
60%
Suspected pattern
solution rates by category
40%
Did not solve
N U h d Sh d0%
20%
None Unshared Shared
29
Visualization Condition
Results: Summary
Hypothesis 1 : ConfirmedPairs of analysts discussed more and performed better when each Pairs of analysts discussed more and performed better when each
partner had half the data than when each partner had all data.
Hypothesis 2 : Partially ConfirmedPairs of analysts with a visualization tool outperformed analysts y p y
without a visualization tool but only if they had split data.
Hypothesis 3: ConfirmedVisualizations benefited collaborative analysis more when each
h d l d h h h h d ll dpartner had split data than when each partner had all data.
30
Design Implications
32
Sharing Traces
Gutwin 2002
33
Gutwin 2002Cleveland, W.S. & Schmieg, G.M. 1987
How Do We Improve Solve Rates?
100%
Percent
80%
60%of Pairs
Solving the Case
60%
40%60%
Case
0%
20% 27%
Split Data All Data0%
Shared Visualization Condition
34
Shared Visualization Condition
Thank You!
National Science Foundation Grant IIS-0325047La ren GieseLauren GieseyMeghan SharmaP l R b iPaul RubritzGail KusbitPeter Scupelli
Image AttributionsChalk body: Johnny Vulcan http://flickr.com/photos/johnnyvulkan/328482660/ Footprint: · ª н м · http://flickr.com/photos/hashir/420913764/Hammer: ZoofyTheJinx http://flickr.com/photos/zoofythejinx/255395476/y J p p y jHandcuffs: pingnews.com http://flickr.com/photos/pingnews/154771251/
35
Questions?
Aruna Balakrishnan [email protected]