Post on 01-Apr-2015
1 Daniel Avrahami - Thesis Proposal
Who, What, and When:Supporting Interpersonal
Communication over Instant
Messaging Daniel Avrahami
Committee:Scott Hudson (Chair)
Eric HorvitzRobert KrautAlon Lavie
2 Daniel Avrahami - Thesis Proposal
Illustration
• John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.
3 Daniel Avrahami - Thesis Proposal
Illustration
• John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.
Since John is pressed for time, he decides to ignore all incoming messages until after he’s done, leaving Anne unable to finish her task.
4 Daniel Avrahami - Thesis Proposal
Illustration (cont)
Consider now if we were able to:• Accurately predict, based on his activity, that John was
not likely to respond to Anne’s message for some time • Predict, based on past communication patterns, that
Anne and John are co-workers
Such models could be used, for example, to increase the salience of the alert, indicating to John that Anne’s message may deserve his immediate attention
5 Daniel Avrahami - Thesis Proposal
Research goals
• The two main goals of my proposed thesis work are to provide a better understanding of factors affecting IM interaction in its context, and to use this understanding for the creation of predictive statistical models and tools that support IM communication.
• In order to achieve these goals my research will use three complementary steps:
6 Daniel Avrahami - Thesis Proposal
Research goals
• Create accurate models that predict responsiveness to incoming IM, and investigate the factors affecting responsiveness (when)
• Investigate the effect of interpersonal relationships on IM interaction, and create statistical models that use this knowledge to predict relationships (who)
• Use basic properties of human dialogue to provide support for balancing of responsiveness and performance (what)
7 Daniel Avrahami - Thesis Proposal
Background
• Instant Messaging, or IM, is one of the most popular communication mediums today
• No longer a medium only for social communication– 12 billion instant messages are sent each day.
Nearly 1 billion messages are exchanged by 28 million business users [IDC Market Analysis’05]
• Useful in many ways: from quick questions and clarifications, coordination and scheduling, to discussions of complex work [Bradner’99; Nardi’00; Handel’02; Herbsleb’02; Isaacs’02]
8 Daniel Avrahami - Thesis Proposal
Background
• Some characteristics of IM:– Sending messages is “cheap”– People can choose when/whether to
respond– Asynchrony means that people can (and
do) multitask [Nardi’00, Isaacs’02]– Can tell whether a receiver is present
• But…
9 Daniel Avrahami - Thesis Proposal
Background
• Especially in the workplace, means that messages may often arrive at inconvenient times
• Presence is not enough
• Unsuccessful communication can have a negative effect on both sender and receiver– Can disrupt the receiver’s work– Can leave the sender waiting for information– True not only for IM
10
Daniel Avrahami - Thesis Proposal
When:Predicting responsiveness
to IM
11
Daniel Avrahami - Thesis Proposal
Background
Want to answer the following question:
• If an instant message were to arrive right now, would the user respond to it? In how long?
12
Daniel Avrahami - Thesis Proposal
How can such models help?
sender receiver
intercept alert mask enhance
awareness
message
13
Daniel Avrahami - Thesis Proposal
sender
How can such models help?
message
receiver
intercept alert mask enhance
14
Daniel Avrahami - Thesis Proposal
sender
How can such models help?
message
receiver
intercept alert mask enhance
15
Daniel Avrahami - Thesis Proposal
sender
How can such models help?
message
receiver
intercept alert mask enhance
16
Daniel Avrahami - Thesis Proposal
sender
How can such models help?
awareness
receiver intercept alert mask enhance
shhhh
17
Daniel Avrahami - Thesis Proposal
sender
How can such models help?
awareness
receiver
intercept alert mask enhance (carefully)
not now
18
Daniel Avrahami - Thesis Proposal
Related work
• Interruptions and disruptions– [Gillie’89 , Cutrell’01 , Hudson’02 , Dabbish’04]
• Interruptibility and cost of interruption– [Horvitz’99 , Horvitz’03, Hudson’03 , Begole’04,
Horvitz’04, Fogarty’05, Iqbal’06]
• Models of presence– [Horvitz’02, Begole’03]
• Responsiveness to Email– [Horvitz’02, Tyler’03]
19
Daniel Avrahami - Thesis Proposal
Data collection
20
Daniel Avrahami - Thesis Proposal
Data collection
• Created a plugin for Trillian Pro (written in C)– Non-intrusive collection of IM and desktop events
– Why this setup?
21
Daniel Avrahami - Thesis Proposal
Data collection (cont)
• What gets collected:
– IM Events• Messages, status changes, etc.
– Desktop Events• Applications, events, etc.
• Each participants records for at least 4 weeks
22
Daniel Avrahami - Thesis Proposal
Data collection (cont.)
• Privacy of data– Masking messages
• for example, the message:“This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”.
• Temporary masking
– Alerting buddies
– Hashing buddy-names
23
Daniel Avrahami - Thesis Proposal
Participants
• 16 participants
– Researchers: 6 full-time employees at an industrial research lab (mean age=40.33)
– Interns: 2 summer interns at the industrial research lab (mean age=34.5)
– Students: 8 Masters students (mean age=24.5)
24
Daniel Avrahami - Thesis Proposal
Participants
• Nearly 5200 hours recorded• Over 90,000 messages• Over 400 buddies• 4 participants provided full text
• On average, participants exchanged a message every: 8.1, 2.2, 3.1 minutes
25
Daniel Avrahami - Thesis Proposal
Responsiveness
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
26
Daniel Avrahami - Thesis Proposal
Responsiveness
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
50%
27
Daniel Avrahami - Thesis Proposal
Responsiveness
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
92%50%
28
Daniel Avrahami - Thesis Proposal
Defining “IM Sessions”
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
session
92%
29
Daniel Avrahami - Thesis Proposal
Defining “Session Initiation Attempts”
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
session
used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes
30
Daniel Avrahami - Thesis Proposal
Features
• For every message:– Features describing IM state. including:
• Day of week• Hour• Is the Message-Window open• Buddy status (e.g., “Away”)• Buddy status duration• Time since msg to buddy• Time since msg from another buddy• Any msg from other in the last 5 minutes• log(time since msg with any buddy)• Is an SIA-5
31
Daniel Avrahami - Thesis Proposal
Features (cont.)
• For every message:– Features describing desktop state (following
Horvitz et al. Fogarty et al. and others). including:• Application in focus• Application in focus duration• Previous application in focus• Previous application in focus duration• Most used application in past m minutes• Duration for most used application in past m minutes• Number of application switches in past m minutes• Amount of keyboard activity in past m minutes• Amount of mouse activity in past m minutes• Mouse movement distance in past m minutes
32
Daniel Avrahami - Thesis Proposal
What are we predicting?
• “Seconds Until Response”– computed, for every incoming message
from a buddy, by noting the time it took until a message was sent to the same buddy
• Examined five responsiveness thresholds– 30 seconds, 1, 2, 5, and 10 minutes
33
Daniel Avrahami - Thesis Proposal
Modeling method
• Weka ML toolkit• Features selected using a wrapper-
based selection technique• AdaBoosting on Decision-Tree models• 10-fold cross-validation
– 10 trials: train on 90%, test on 10%– Next we report combined accuracy
34
Daniel Avrahami - Thesis Proposal
79.883.8
87.089.4 90.1
0
10
20
30
40
50
60
70
80
90
100
30sec 1min 2min 5min 10min
Predict response within
% A
ccu
rate
Results (full feature-set models)
All significantly better than the prior probability (p<.001)
35
Daniel Avrahami - Thesis Proposal
Results (buddy-independent models)
• Previous models used information about the buddy (e.g., time since messaging that buddy)
• Can predict different responsiveness for different buddies– But what if you wanted just one level of
responsiveness?
• Built models that did not use any buddy-related features
36
Daniel Avrahami - Thesis Proposal
79.882.5
87.0 89.4 89.3
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
0
10
20
30
40
50
60
70
80
90
100
30sec 1min 2min 5min 10min
Predict response within
% A
ccu
rate
Results (buddy-independent models)
all significantly better than the prior probability (p<.001) BUT not sig. diff. from previous set
37
Daniel Avrahami - Thesis Proposal
Not using content is good and bad
• Pros:– Privacy preserving– Easy
• Cons:– Misinterpreting quick responses– Misinterpreting messages that do not need
responses– > Can lead to some (machine) learning
errors
38
Daniel Avrahami - Thesis Proposal
Planned work: Content
• Need to be able to answer the following question (when a incoming message arrives):
Is this message the beginning of a new session or part of the previous session?
• I plan to develop a method for answering this question automatically using content analysis
39
Daniel Avrahami - Thesis Proposal
Planned work: Content
40
Daniel Avrahami - Thesis Proposal
Planned work: Content
• Evaluation plans:
– Obtain manual coding of a random subset of segment boundaries identified as correct or incorrect
– Use inter-coder agreement to establish a performance benchmark
– Compare to accuracy of automated approach
41
Daniel Avrahami - Thesis Proposal
Planned work: Probabilities
• Create a probabilistic model predicting the likelihood of the arrival of the next message in a session– Given evidence about the session so far– What is the probability of
• Another message within time T ?• Another message within time >= T ?
• Useful in identifying ends of sessions– Good for load balancing (help-desk
scenarios)
42
Daniel Avrahami - Thesis Proposal
Planned work: Understanding
• Showed the successful creation of statistical models that predict responsiveness
• A better understanding is needed of the connection between responsiveness and a person’s communication- and work-context
43
Daniel Avrahami - Thesis Proposal
Planned work: Understanding
• I plan to investigate in detail the contribution of specific features and the interactions between those features to responsiveness
– Preliminary results suggest, for example, that indications of work fragmentation significantly affect responsiveness
44
Daniel Avrahami - Thesis Proposal
Who:Relationships and
communication patterns
45
Daniel Avrahami - Thesis Proposal
Relationships and IM communication
• People use IM for both work and social communication
• Availability might depend on relationship
• Wanted to investigate the effect of relationship on basic communication patterns
46
Daniel Avrahami - Thesis Proposal
Background
• Relationship type has significant effects on communication, including the quality, purpose and perceived value [Duck’91]
• Cues, such as tempo, pauses, speech rates and the frequency of turns, affect the way in which conversation partners perceive each other [Feldstein’94]
• Frequency affects communication [FTF:Whittaker’94, IM:Isaacs’02]
47
Daniel Avrahami - Thesis Proposal
Buddy Coder
• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]
48
Daniel Avrahami - Thesis Proposal
• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]
Buddy Coder
49
Daniel Avrahami - Thesis Proposal
• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]
Buddy Coder
50
Daniel Avrahami - Thesis Proposal
• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]
Buddy Coder
51
Daniel Avrahami - Thesis Proposal
• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]
Buddy Coder
52
Daniel Avrahami - Thesis Proposal
Relationships distribution
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Researchers Interns Students
Work
Mix
Social
Other
53
Daniel Avrahami - Thesis Proposal
Session-level measures
54
Daniel Avrahami - Thesis Proposal
Session-level measures
# Time Message Text
1 17:42:45 B: Hey [Participant’s name]
2 17:42:56 B: what time does your group get in the AM?
3 17:42:57 P: hey
4 17:43:01 P: usually around 10
5 17:43:25 B: ok
6 17:43:38 B: i want to start circulating the card in the AM
7 17:43:58 P: ok, good idea
8 17:44:02 P: that's for coordinating this
9 17:44:13 B: no problem
10 17:44:27 P: thanks :-)
11 17:44:35 P: sorry bout the typo
12 17:44:38 B: is ok
55
Daniel Avrahami - Thesis Proposal
Session-level measures
# Time Message Text
1 17:42:45 B: Hey [Participant’s name]
2 17:42:56 B: what time does your group get in the AM?
3 17:42:57 P: hey
4 17:43:01 P: usually around 10
5 17:43:25 B: ok
6 17:43:38 B: i want to start circulating the card in the AM
7 17:43:58 P: ok, good idea
8 17:44:02 P: that's for coordinating this
9 17:44:13 B: no problem
10 17:44:27 P: thanks :-)
11 17:44:35 P: sorry bout the typo
12 17:44:38 B: is ok
Variable Value
Group Student
Relationship Work
Duration 1.88 minutes
Message Count 12
Turn Count 7
Character Count 232
Messages per Minute 6.4
Messages per Turn 1.71
Characters per Message 19.3
Seconds Until First Reply 1 seconds
Minimum Gap (between turns) 1 seconds
Maximum Gap (between turns) 24 seconds
Average Gap (between turns) 12.2 seconds
Time of Day 5:44 pm
56
Daniel Avrahami - Thesis Proposal
The effect of relationships
• Used a repeated-measures ANOVA
– Relationship Category (Work, Mix, Social) and Group (Researchers, Interns, Students) were repeated
– Participants and BuddyID modeled as random effects
– Participants nested in Group– BuddyID nested first in Participants, then in Group
– N = 3297
57
Daniel Avrahami - Thesis Proposal
Results
58
Daniel Avrahami - Thesis Proposal
Summary of Results
• Sessions with Social contacts were longer and with more messages BUT at a significantly slower pace– Maybe giving less attention to these
sessions?
• Sessions with Work contacts were at a faster pace with longer messages– Grounding? Complex concepts?
59
Daniel Avrahami - Thesis Proposal
Results: Session length
• Significant effect on Session Duration (p<.001)– Social significantly longer sessions than
both Mix and Work (Work and Mix n.s.)
• Similar effects for– Number of Turns
Number of MessagesNumber of Characters• Duration correlated at >.85
0
1
2
3
4
5
6
7
8
w ork mix social
Relationship
Du
rati
on
(m
inu
tes)
60
Daniel Avrahami - Thesis Proposal
Results: Messaging rate
• Significant effect on Messaging Rate (p<.01)– Social significantly slower than Mix (p=.003)– Social marginally slower than Work (p=.078)
• Maximum-Gap (p<.05)Social longer than Work(p=.013)
0
1
2
3
4
5
6
7
w ork mix social
Relationship
Mes
sag
es-p
er-M
inu
te
61
Daniel Avrahami - Thesis Proposal
Results: Length of messages
• Significant effect on Message Length (Characters-per-Message) (p<.001)– Work significantly longer than both Social
(p<.001) and Mix (p=.002)
0
5
10
15
20
25
30
35
40
45
w ork mix social
Relationship
Ch
arac
ters
-per
-Mes
sag
e
62
Daniel Avrahami - Thesis Proposal
Predicting relationships
63
Daniel Avrahami - Thesis Proposal
Predicting relationships
• How can it be used?– Augmenting IM systems
• Indicators of unavailability• Differential alerts
– Shared with other mediums• E.g. Email
– Provide organizational overview
64
Daniel Avrahami - Thesis Proposal
0 (Work)
0 (Work)
1 (Social)
1 (Social)
0 (Work)b1i
b1
b1
b2
b2
0 (Work)
0 (Work)
1 (Social)
0 (Work)
1 (Social)
1 (Social)
0 (Work)
0 (Work)
0 (Work)
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >i
i
i
i
SN Buddy
b1i 3 .333 Yes
2 .5 Nob2i
SN nBuddy
ActualSessionVariables
Predicting relationships
Cross-validation with 16 models (omitting one participant each time)
Nominal Logistic Regression
65
Daniel Avrahami - Thesis Proposal
Models performance
• Results from pairs with 2 sessions or more (78% of the data)
Classified as
Work Social
Work40.9%(83)
5.9%(12)
Social14.8%(30)
38.4%(78)
Accuracy: 79.3%
66
Daniel Avrahami - Thesis Proposal
Models performance
• Results from pairs with 2 sessions or more (78% of the data)
• Both significantly better than the prior probability
Classified as
Work Social
Work40.9%(83)
5.9%(12)
Social14.8%(30)
38.4%(78)
Accuracy: 79.3%
Classified as
Work Mix Social
Work25.3%(74)
5.1%(15)
2.0%(6)
Mix8.2%(24)
14.7%(43)
7.8%(23)
Social9.6%(28)
17.1%(50)
10.2%(30)
Overall Accuracy: 50.2%Work vs. Rest: 75.1%Social vs. Rest: 63.5%
67
Daniel Avrahami - Thesis Proposal
Additional work
• Find a way to distinguish Mix from Social
• Examine the effects of additional aspects of relationships on communication– The effects of physical distance– The effects of buddy-familiarity– The effects of task criticality
68
Daniel Avrahami - Thesis Proposal
What:Using content to balance
responsiveness and performance
69
Daniel Avrahami - Thesis Proposal
Responsiveness / performance tradeoff
• Users often multitask when using instant messaging [Nardi’00, Isaacs’02, Voida’02]
• Users often have to choose between– Staying on task and being responsive to IM
buddies
• Current solutions typically force users to choose one or the other:– Update ‘away’ messages– Turn off IM client
70
Daniel Avrahami - Thesis Proposal
•Quick response - “do you have the figures I need?”
•Leisurely response - “check out www.cnn.com”
•Politely deferred - “ru busy?”
•No response - “going to meeting. ttyl”
Expectations for responsiveness
71
Daniel Avrahami - Thesis Proposal
The approach: QnA
• Users ignore, to the best of their ability, the alerts of incoming messages – Transitioning (internally) to being unavailable
• By observing the content of messages, QnA automatically highlights incoming messages that may deserve their attention– In particular, potential questions and answers
72
Daniel Avrahami - Thesis Proposal
Demo
• (Good luck with the demo)
73
Daniel Avrahami - Thesis Proposal
Why questions and answers?
• A question and an answer form an ‘Adjacency pair’ (Schegloff & Sacks’73)
• From “Arenas of Language Use” by Clark (1992)“Given a first pair part, a second pair part is conditionally relevant, that is, relevant and expectable, as the next utterance.
Once A has asked the question, it is relevant and expectable for B to answer in the next turn.” (p. 157)
74
Daniel Avrahami - Thesis Proposal
• QnA listens to incoming and outgoing messages– when an outgoing messages is sent
• if it is a question– remember that expecting a response
– when an incoming messages arrives• if it is a question and/or we are expecting an
answer– wait x seconds to see if user attends to the message– if did not attend then show QnA notification
How does it work?
75
Daniel Avrahami - Thesis Proposal
is_a_question?
• Match to list of questions that can be ‘politely deferred’– (are|r) (you|u) there– busy?
• Go through list of rules and look for match– (?|/) at end of sentence– what (is|are|r|were|does|do|did|should|can)– did(|n’t|nt) (i|u|you|he|she|they|we)– (are|r) (you|u)– huh
76
Daniel Avrahami - Thesis Proposal
Delaying notifications
• First determine whether the user is engaged in the conversation by listening to these events:– Opens the message window– Clicks on the message window– Types in the message window– Message window was in focus when message
arrived
• Multiple questions from same buddy within same delay period will only get one alert
77
Daniel Avrahami - Thesis Proposal
Issues
• Determining that a message contains a question or an answer can be difficult– interleaved conversations– many short messages that comprise a single
turn– loose grammar and spelling
• Gives buddies a way to increase the salience of their messages. what if they abuse it?
78
Daniel Avrahami - Thesis Proposal
QnA summary
• QnA: A tool that allows users to stay on task, but still seem responsive to buddies who expect it
• Allows users to transition between work modes– Sits quietly in the background when the
user attends to messages– Only notifies when the user ignores
messages
79
Daniel Avrahami - Thesis Proposal
Future work
• Collect feedback from users– A few users who have used QnA for over 2
years now– But would like more users
• Please download QnA from my homepage
• Improve question identification
• Implement ‘ignore list’
80
Daniel Avrahami - Thesis Proposal
Conclusions
81
Daniel Avrahami - Thesis Proposal
Conclusions
• I have presented planned and completed work on analysis and generation of predictive modeling in support of interpersonal communication over IM:– Completed work along with additional planned
work on predictions of responsiveness to IM– Completed work on analysis and predictions of
interpersonal relationships and their effect on communication
– Completed work on the use of basic properties of human dialogue to allow users to balance responsiveness and performance
82
Daniel Avrahami - Thesis Proposal
Contributions
• This work’s contribution to the HCI field will span both theoretical and applied aspects. – From a theoretical point of view, this work will
provide insights into the factors that influence interpersonal communication patterns and responsiveness.
– At the applied level, this work will provide predictive statistical models that can be used in many applications.
– Finally, this work promotes the creation of tools that use knowledge and predictive models generated from naturally occurring interaction.
83
Daniel Avrahami - Thesis Proposal
Acknowledgements
• No acknowledgements until my defense!
• Thank you Thi!
• Good luck to Darren, James, Jeff and Laura
84
Daniel Avrahami - Thesis Proposal
Thank you.
This work was funded in part by the National Science Foundation under grants IIS 0121560 and IIS
0325351 andthis material is based upon work supported by the Defense
Advanced Research Projects Agency (DARPA) under Contract No. NBCHD030010