1 Daniel Avrahami – Doctoral Symposium – UIST06 Who, What, and When: Supporting Interpersonal...

1

Daniel Avrahami – Doctoral Symposium – UIST’06

Who, What, and When:Supporting Interpersonal Communication over Instant Messaging

Daniel AvrahamiCarnegie Mellon University

www.cs.cmu.edu/~nx6

2


Illustration

John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.

3


Illustration

John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.

Since John is pressed for time, he decides to ignore all incoming messages until after he’s done, leaving Anne unable to finish her task.

4


Illustration (cont)

Consider now if we were able to: Accurately predict, based on his activity, that John

was not likely to respond to Anne’s message for some time

Predict, based on past communication patterns, that Anne and John are co-workers

Such models could be used, for example, to increase the salience of the alert, indicating to John that Anne’s message may deserve his immediate attention

5


Research goals

The two main goals of my research work are to provide a better understanding of factors affecting IM interaction in its context, and to use this understanding for the creation of predictive statistical models and tools that support IM communication.

In order to achieve these goals my research will use three complementary steps:

6


Research goals

Create accurate models that predict responsiveness to incoming IM, and investigate the factors affecting responsiveness (when)

Investigate the effect of interpersonal relationships on IM interaction, and create statistical models that use this knowledge to predict relationships (who)

Use basic properties of human dialogue to provide support for balancing of responsiveness and performance (what)

7


Background

Instant Messaging, or IM, is one of the most popular communication mediums today 12 billion instant messages are sent each day. Nearly 1

billion messages are exchanged by 28 million business users [IDC Market Analysis’05]

Useful in many ways: from quick questions and clarifications, coordination and scheduling, to discussions of complex work [Bradner’99; Nardi’00; Handel’02; Herbsleb’02; Isaacs’02]

IM has a number of shortcomings (Asynchrony + Limited aware + Low cost for sending)

8


When:Predicting responsiveness to IM

[Presented at CHI’06]

9


Background

Wanted to answer the following question:

If an instant message were to arrive right now, would the user respond to it? In how long?

10


Data Collection

11


Data collection

Created a plugin for Trillian Pro (written in C) Non-intrusive collection of:

IM events desktop events

12


Participants

16 participants to date Nearly 5200 hours recorded Over 90,000 messages Over 400 buddies 4 participants provided full text

On average, participants exchanged a message every: 3.4 minutes (researchers avg=8.1)

13


Responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

92%

14


Defining “Session Initiation Attempts”

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

session

used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes

92%

15


What are we predicting?

Generate a set of features for every message: IM state and desktop state

“Seconds Until Response” computed, for every incoming message from a

buddy, by noting the time it took until a message was sent to the same buddy

Examined five responsiveness thresholds 30 seconds, 1, 2, 5, and 10 minutes

16


Modeling method

Weka ML toolkit Features selected using a wrapper-based

selection technique AdaBoosting on Decision-Tree models 10-fold cross-validation

10 trials: train on 90%, test on 10% Next I report combined accuracy

17


79.883.8

87.089.4 90.1

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

Results (full feature-set models)

All significantly better than the prior probability (p<.001)

(Graph shows 5-minute

subset only)

18


Results (buddy-independent models)

Previous models used information about the buddy (e.g., time since messaging that buddy)

Can predict different responsiveness for different buddies But what if you wanted just one level of

responsiveness?

Built models that did not use any buddy-related features

19


79.882.5

87.0 89.4 89.3

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

Results (buddy-independent models)

all significantly better than the prior probability (p<.001) BUT not sig. diff. from previous set

20


Some practical considerations

Preserving plausible deniability

Making predictions about the receiver, visible to the receiver

Multiple concurrent levels of responsiveness

21


Who:Relationships and IM Communication

[To be presented at CSCW’06]

22


Relationships and IM communication

People use IM for both work and social communication

Prior research shows that relationship type has significant effect on fact-to-face and other voice communication (Duck’91)

Wanted to investigate the effect of relationship on basic communication patterns

23


Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused]

Buddy Coder

24


Session-level measures

# Time Message Text

1 17:42:45 B: Hey [Participant’s name]

2 17:42:56 B: what time does your group get in the AM?

3 17:42:57 P: hey

4 17:43:01 P: usually around 10

5 17:43:25 B: ok

6 17:43:38 B: i want to start circulating the card in the AM

7 17:43:58 P: ok, good idea

8 17:44:02 P: that's for coordinating this

9 17:44:13 B: no problem

10 17:44:27 P: thanks :-)

11 17:44:35 P: sorry bout the typo

12 17:44:38 B: is ok

Variable Value

Group Student

Relationship Work

Duration 1.88 minutes

Message Count 12

Turn Count 7

Character Count 232

Messages per Minute 6.4

Messages per Turn 1.71

Characters per Message 19.3

Seconds Until First Reply 1 seconds

Minimum Gap (between turns) 1 seconds

Maximum Gap (between turns) 24 seconds

Average Gap (between turns) 12.2 seconds

Time of Day 5:44 pm

25


The effect of relationships

Used a repeated-measures ANOVA Relationship Category (Work, Mix, Social) and

Group (Researchers, Interns, Students) were repeated

Participants and BuddyID modeled as random effects

Participants nested in Group BuddyID nested first in Participants, then in

Group

N=3297 sessions

26


Results

27


Summary of Results

Sessions with Social contacts were longer and with more messages BUT at a significantly slower pace Maybe giving less attention to these sessions?

Sessions with Work contacts were at a faster pace with longer messages Grounding? Complex concepts?

28


Results: Session length

Significant effect on Session Duration (p<.001)

Social significantly longer sessions than both Mix and Work (Work and Mix n.s.)

Similar effects forNumber of TurnsNumber of MessagesNumber of Characters(Duration correlated at >.85)

0

1

2

3

4

5

6

7

8

w ork mix social

Relationship

Du

rati

on

(m

inu

tes)

29


Results: Messaging rate

Significant effect on Messaging Rate (p<.01) Social significantly slower than Mix (p=.003) Social marginally slower than Work (p=.078)

Maximum-Gap (p<.05)Social longer than Work(p=.013)

0

1

2

3

4

5

6

7

w ork mix social

Relationship

Mes

sag

es-p

er-M

inu

te

30


Results: Length of messages

Significant effect on Message Length (Characters-per-Message) (p<.001) Work significantly longer than both Social (p<.001) and Mix

(p=.002)

0

5

10

15

20

25

30

35

40

45

w ork mix social

Relationship

Ch

arac

ters

-per

-Mes

sag

e

31


Predicting relationships

32



How can it be used? Augmenting IM systems

Indicators of unavailability Differential alerts

Shared with other mediums E.g. Email

Provide organizational overview

33


Models performance

2-step Logistic regression model with 16-fold cross validation

Results from pairs with 2 sessions or more (78% of the data) Significantly better than the prior probability

Classified as

Work Social

Work40.9%(83)

5.9%(12)

Social14.8%(30)

38.4%(78)

Accuracy: 79.3%

Classified as

Work Mix Social

Work25.3%(74)

5.1%(15)

2.0%(6)

Mix8.2%(24)

14.7%(43)

7.8%(23)

Social9.6%(28)

17.1%(50)

10.2%(30)

Overall Accuracy: 50.2%Work vs. Rest: 75.1%Social vs. Rest: 63.5%

34


What:Using content to balance

responsiveness and performance

[Presented at CSCW’04]

35


Responsiveness / Performance Tradeoff

Users often multitask when using instant messaging [Nardi’00, Isaacs’02, Voida’02]

Users often have to choose between Staying on task and being responsive to IM

Current solutions typically force users to choose one or the other: Update ‘away’ messages Turn off IM client

36


Quick response - “do you have the figures?”

Leisurely response - “check out www.cnn.com”

Politely deferred - “ru busy?”

No response - “going to meeting. ttyl”

Expectations for responsiveness

37


The approach: QnA

Users ignore, to the best of their ability, the alerts of incoming messages Transitioning (internally) to being unavailable

By observing the content of messages, QnA automatically highlights incoming messages that may deserve their attention In particular, potential questions and answers

38


Why questions and answers?

A question and an answer form an ‘Adjacency pair’ (Schegloff & Sacks’73)

From “Arenas of Language Use”“Given a first pair part, a second pair part is conditionally relevant, that is, relevant and expectable, as the next utterance.

Once A has asked the question, it is relevant and expectable for B to answer in the next turn.”

(Clark 1992, p. 157)

39


How does it work?

QnA listens to incoming and outgoing messages when an outgoing messages is sent

if it is a question remember that expecting a response

when an incoming messages arrives if it is a question and/or we are expecting an answer

wait x seconds to see if user attends to the message if did not attend then show QnA notification

40


is_a_question?

Match to list of questions that can be ‘politely deferred’ (are|r) (you|u) there busy?

Go through list of rules and look for match (?|/) at end of sentence what (is|are|r|were|does|do|did|should|can) did(|n’t|nt) (i|u|you|he|she|they|we) (are|r) (you|u) huh

41


QnA summary

QnA: A tool that allows users to stay on task, but still seem responsive to buddies who expect it

Allows users to transition between work modes Sits quietly in the background when the user

attends to messages Only notifies when the user ignores messages

Download from www.cs.cmu.edu/~nx6

42


Conclusions & Future Work

43


Conclusions

I have presented work on analysis and generation of predictive modeling in support of interpersonal communication over IM:

Work on predictions of responsiveness to IM communication (specifically to session initiation attempts)

Work on analysis and predictions of interpersonal relationships and their effect on communication

Work on the use of basic properties of human dialogue to allow users to balance responsiveness and performance

44


Current and Planned Work

Understanding Responsiveness Investigate in detail the contribution of specific features

Investigate distribution of responsiveness over time

Content Analysis Determining the Communication Goals Content-based transcript segmentation

45


this work was funded in part by NSF Grants IIS-0121560, IIS-0325351, and by DARPA Contract No. NBCHD030010

thank you

for more info visit: www.cs.cmu.edu/~nx6

or email: [email protected]

46


Process diagram

47


Process diagram

48


Process diagram

49


Process diagram

50


Related work

Interruptions and disruptions [Gillie’89 , Cutrell’01 , Hudson’02 , Dabbish’04]

Interruptibility and cost of interruption [Horvitz’99 , Horvitz’03, Hudson’03 , Begole’04, Horvitz’04,

Fogarty’05, Iqbal’06]

Models of presence [Horvitz’02, Begole’03]

Responsiveness to Email [Horvitz’02, Tyler’03]

51


Participants

16 participants to date

Researchers: 6 full-time employees at an industrial research lab (mean age=40.33)

Interns: 2 summer interns at the industrial research lab (mean age=34.5)

Students: 8 Masters students (mean age=24.5)

52


How can such models help?

sender receiver

intercept alert mask enhance

awareness

message

53


sender


message

receiver


54


sender


message

receiver


55


sender


message

receiver


56


sender


awareness

receiver intercept alert mask enhance

shhhh

57


sender


awareness

receiver

intercept alert mask enhance (carefully)

not now

58


Data collection (cont.)

Privacy of data Masking messages

for example, the message:“This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”.

Temporary masking

Alerting buddies

Hashing buddy-names

59


Who:Relationships and IM Communication

[To be presented at CSCW’06]

60


Relationships and IM communication

People use IM for both work and social communication

Availability might depend on relationship

Wanted to investigate the effect of relationship on basic communication patterns

61


Background

Relationship type has significant effects on communication, including the quality, purpose and perceived value [Duck’91]

Cues, such as tempo, pauses, speech rates and the frequency of turns, affect the way in which conversation partners perceive each other [Feldstein’94]

Frequency affects communication [FTF:Whittaker’94, IM:Isaacs’02]

62


Relationships distribution

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Researchers Interns Students

Work

Mix

Social

Other

63


Results

Relationship Category

Work Mix Social Analysis of Variance

Variables Mean StdErr Mean StdErr Mean StdErr F d.f. p

Duration (in minutes) 4.0 0.6 5.2 0.6 6.6 0.5 8.04 2/331 <.001

Message count 13.8 3.0 19.8 3.1 25.9 2.8 6.11 2/398 <.01

Turn count 8.8 1.7 12.2 1.7 15.3 1.6 5.96 2/374 <.01

Character count 459.5 122.7 673.6 123.6 844.6 115.2 4.71 2/340 <.01

Messages-per-Minute 6.0 0.5 6.2 0.4 4.6 0.4 4.75 2/99 <.05

Messages-per-Turn § 1.5 0.05 1.5 0.05 1.6 0.05 2.32 2/312

Characters-per-Message 37.9 2.5 31.5 2.5 30.1 2.4 7.85 2/229 <.001

Seconds Until First Reply 36.9 3.0 35.0 3.1 36.0 2.7 0.11 2/151

Minimum Gap (between turns) 12.0 1.8 12.4 1.9 12.1 1.6 0.02 2/111

Maximum Gap (between turns) 68.7 3.8 77.0 3.9 81.8 3.4 3.25 2/173 <.05

Average Gap (between turns) 28.8 2.2 28.3 2.3 29.2 2.0 0.10 2/181

Time of Day § 14.6 0.4 14.6 0.4 14.7 0.4 0.04 2/253

64


0 (Work)

0 (Work)

1 (Social)

1 (Social)

0 (Work)b1i

b1

b1

b2

b2

0 (Work)

0 (Work)

1 (Social)

0 (Work)

1 (Social)

1 (Social)

0 (Work)

0 (Work)

0 (Work)

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >i

i

i

i

SN Buddy

b1i 3 .333 Yes

2 .5 Nob2i

SN nBuddy

ActualSessionVariables


Cross-validation with 16 models (omitting one participant each time)

Nominal Logistic Regression

65


What:Using content to balance

responsiveness and performance

[Presented at CSCW’04]

66


Issues

Determining that a message contains a question or an answer can be difficult interleaved conversations many short messages that comprise a single turn loose grammar and spelling

Gives buddies a way to increase the salience of their messages. what if they abuse it?

67


Future work

Collect feedback from users A few users who have used QnA for over 2 years

now But would like more users

Please download QnA from my homepage

Improve question identification

Implement ‘ignore list’

68


Conclusions & Future Work

69


Contributions

This work’s contribution to the HCI field will span both theoretical and applied aspects. From a theoretical point of view, this work will

provide insights into the factors that influence interpersonal communication patterns and responsiveness.

At the applied level, this work will provide predictive statistical models that can be used in many applications.

Finally, this work promotes the creation of tools that use knowledge and predictive models generated from naturally occurring interaction.

1 Daniel Avrahami – Doctoral Symposium – UIST06 Who, What, and When: Supporting Interpersonal...

Documents

Transcript of 1 Daniel Avrahami – Doctoral Symposium – UIST06 Who, What, and When: Supporting Interpersonal...