Pitfalls of Information Access with Visualizationsin Remote Collaborative Analysis

Aruna D. Balakrishnan, Susan R. Fussell, Sara Kiesler & Aniket KitturHuman Computer Interaction InstituteCarnegie Mellon UniversityCarnegie Mellon University

February 10, 2010

Synchronous Remote Collaborations

di ib d dH d fi distributed datainformation overload

How do you figure out what’s important?

3

Visualizations

textual & numerical data

visualizations

4

visualizations

Interactive Social Network Diagrams

Larkin & Simon 1987Sparrow1998Viégas, Wattenberg, & Dave 2004

5

g , g,Balakrishnan, Fussell, & Kiesler 2008

Shared Information

BenefitsEliminates need f r data e chan eEliminates need for data exchangeFocus on understanding & analysis of materialsSh d l d l Shared mental model (Fraidin 2004; Blockeel & Moyle 2002; Kozlowski, Ilgen & Klimoski 2006)

DisadvantagesC fi i bi Confirmation bias (Mojzisch & Schulz-Hardt 2005)

6

Distributed Information

BenefitsIncreased disc ssi n d e t f rced inf e chan eIncreased discussion due to forced info exchangeMultiple perspectives

DisadvantagesForced information exchange

7

Hypotheses

Hypothesis 1Pairs of analysts will perform better when each partner has partial Pairs of analysts will perform better when each partner has partial

data than when each partner has all the data.

Hypothesis 2 Pairs of analysts with a visualization tool will outperform analysts

without a visualization tool.

H h i 3Hypothesis 3Visualizations will benefit collaborative analysis more when each

partner has partial data than when each partner has all the datapartner has partial data than when each partner has all the data.

8

Experiment Design

Detective mystery experimental paradigm Pairs of participants investigated multiple crimes (Scupelli et. al, 2005)p p g p ( p , )

Shared task To find a serial killer responsible for several of these crimesTo find a serial killer responsible for several of these crimes

Distributed evidence E id f i l kill di t ib t d 7 Evidence for serial killer distributed among 7 casesOnly 5 of 7 cases had relevant information to the task

R ll b iRemote collaborationSeated apart to simulate distributed workAccess to Instant MessengerAccess to Instant Messenger

10

Experiment Task

11

Experiment Design

Between-subjects

2 x 3 factorial designPairs randomly assigned to

1 of 2 information access conditions1 of 3 visualization conditions1 of 3 visualization conditions

12

Experimental ConditionsVisualization Conditions

Shared VisUnshared VisNo VisInfo Access Conditions

Split Data

N = 15 N = 15 N = 15

All All Data N = 15N = 15 N = 15

13

No Visualization

Spreadsheet Case file

IM

15

IM

Unshared Visualization

Visualization Case file

IM

16

IM

Shared Visualization

Case fileVisualization

IM

17

IM

Experiment Details

180 ParticipantsAge range from 18 64 yearsAge range from 18 – 64 yearsFluency in English required

Procedure30 minutes

Informed consentTrained in detective work Trained in visualization useTrained in visualization use

60 minutesSerial Killer task

18

Dependent Measures

S l ti d d T k P f Solutions recorded in final reports

Task PerformanceSolve serial killer task

C mm nicati nInstant messenger

logs

CommunicationAmount of communicationTypes of communication gypes o co u cat o(intercoder reliability kappa = .71)

19

Results: Average number of IM words

500

600 Split DataAll Data

400

500 All Data

Average # IM

200

300# IM words

per

0

100

N U h d Sh d

person

None Unshared Shared

( 0 0 )

21

Visualization Condition(p < 0.01)

Results: Split Data Pair Solve Rate

100%Split Data

Percentof Pairs

80%

60%of Pairs Solving

the Case40%

N U h d Sh d0%

20%

None Unshared Shared

22

Visualization Condition

Results: All Data Pair Solve Rate

100%Split DataAll Data

Percentof Pairs

80%

60%

All Data

of Pairs Solving

the Case40%

N U h d Sh d0%

20%

None Unshared Shared

23

Visualization Condition

Results: Communication

# Hypotheses SharedDiscussion of a new hypothesis regarding the serial killer case

I think these four blunt instrument victims areinstrument victims areconnected.

I feel like it is a suspicious man on the b sbus.

24

Results: Hypotheses Discussion

4.0Split DataAll Data

3.0

hese

s se

d rs

on

All Data

1 0

2.0

Hyp

oth

disc

uspe

r pe

r

0.0

1.0

N U h d Sh dNone Unshared Shared

25

Visualization Condition

Results: Correct and Incorrect Solutions

Solved100%

Split All Split All Split All

Incorrect SuspectIndividual

solution

80%

60%

Suspected pattern

solution rates by category

40%

Did not solve

N U h d Sh d0%

20%

None Unshared Shared

26

Visualization Condition

Results: Correct and Incorrect Solutions

Solved100%

Split All Split All Split All

Incorrect SuspectIndividual

solution

80%

60%

Suspected pattern

solution rates by category

40%

Did not solve

N U h d Sh d0%

20%

None Unshared Shared

27

Visualization Condition

Results: Correct and Incorrect Solutions

Solved100%

Split All Split All Split All

confirmation bias

Incorrect SuspectIndividual

solution

80%

60%

Suspected pattern

solution rates by category

40%

Did not solve

N U h d Sh d0%

20%

None Unshared Shared

28

Visualization Condition

Results: Correct and Incorrect Solutions

Solved100%

Split All Split All Split All

Incorrect SuspectIndividual

solution

80%

60%

Suspected pattern

solution rates by category

40%

Did not solve

N U h d Sh d0%

20%

None Unshared Shared

29

Visualization Condition

Results: Summary

Hypothesis 1 : ConfirmedPairs of analysts discussed more and performed better when each Pairs of analysts discussed more and performed better when each

partner had half the data than when each partner had all data.

Hypothesis 2 : Partially ConfirmedPairs of analysts with a visualization tool outperformed analysts y p y

without a visualization tool but only if they had split data.

Hypothesis 3: ConfirmedVisualizations benefited collaborative analysis more when each

h d l d h h h h d ll dpartner had split data than when each partner had all data.

30

Design Implications

32

Sharing Traces

Gutwin 2002

33

Gutwin 2002Cleveland, W.S. & Schmieg, G.M. 1987

How Do We Improve Solve Rates?

100%

Percent

80%

60%of Pairs

Solving the Case

60%

40%60%

Case

0%

20% 27%

Split Data All Data0%

Shared Visualization Condition

34

Shared Visualization Condition

Thank You!

National Science Foundation Grant IIS-0325047La ren GieseLauren GieseyMeghan SharmaP l R b iPaul RubritzGail KusbitPeter Scupelli

Image AttributionsChalk body: Johnny Vulcan http://flickr.com/photos/johnnyvulkan/328482660/ Footprint: · ª н м · http://flickr.com/photos/hashir/420913764/Hammer: ZoofyTheJinx http://flickr.com/photos/zoofythejinx/255395476/y J p p y jHandcuffs: pingnews.com http://flickr.com/photos/pingnews/154771251/

35

Questions?

Aruna Balakrishnan aruna@cs.cmu.edu