Responsiveness in IM: Predictive Models Supporting Inter-Personal Communication

Daniel Avrahami, Scott E. HudsonCarnegie Mellon University

www.cs.cmu.edu/~nx6

Q: if an instant message were to arrive right now, would the user respond to it? in how long?

collected field data 5200 hours 90,000 messages IM and desktop

events

models predicting responsiveness as high as 90.1%

why should we care?

why should we care?

IM is one of the most popular communication mediums no longer a medium just for kids (work / parents)

sending messages is “cheap” but the potential for interruptions is great

unsuccessful communication can have a negative effect on both sender and receiver can disrupt the receiver’s work can leave the sender waiting for information true not only for IM

how can such models help?

sender receiver

intercept alert mask enhance

awareness

message

sender

how can such models help?

message

receiver

intercept alert mask enhance

sender

how can such models help?

message

receiver

intercept alert mask enhance

sender

how can such models help?

message

receiver

intercept alert mask enhance

sender

how can such models help?

awareness

receiver intercept alert mask enhance

shhhh

sender

how can such models help?

awareness

receiver

intercept alert mask enhance (carefully)

not now

related work

instant messaging [Nardi’00 , Isaacs’02 , Voida’02]

interruptions and disruptions [Gillie’89 , Cutrell’01 , Hudson’02 , Dabbish’04]

models of presence and interruptibility [Horvitz’02 , Begole’02 , Hudson’03 , Begole’04, Horvitz’04 ,

Fogarty’05 , Iqbal’06]

coming up…

data collection participants responsiveness overview predictive models

how (features and classes) results

a closer look (new! not in the paper) future work

data collection

a plugin for Trillian Pro (written in C) non-intrusive collection of IM and desktop events

data collection (cont.)

privacy of data masking messages

for example, the message “This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”.

alerting buddies hashing buddy-names

4 participants provided full content

participants

16 participants Researchers: 6 full-time employees at an

industrial research lab (mean age=40.33) Interns: 2 summer interns at the industrial

research lab (mean age=34.5) Students: 8 Masters students (mean age=24.5)

nearly 5200 hours recorded over 90,000 messages

responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

50%

responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

92%50%

defining “IM Sessions”

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

session

92%

defining “Session Initiation Attempts”

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes

session

features

for every message: features describing IM state. including:

Day of week Hour Is the Message-Window open Buddy status (e.g., “Away”) Buddy status duration Time since msg to buddy Time since msg from another buddy Any msg from other in the last 5 minutes log(time since msg with any buddy) Is an SIA-5

features (cont.)

for every message: features describing desktop state (following Horvitz et al.

Fogarty et al. and others). including: Application in focus Application in focus duration Previous application in focus Previous application in focus duration Most used application in past m minutes Duration for most used application in past m minutes Number of application switches in past m minutes Amount of keyboard activity in past m minutes Amount of mouse activity in past m minutes Mouse movement distance in past m minutes

what are we predicting?

“Seconds until Response” computed, for every incoming message from a

buddy, by noting the time it took until a message was sent to the same buddy

examined five responsiveness thresholds 30 seconds, 1, 2, 5, and 10 minutes

modeling method

features selected using a wrapper-based selection technique

AdaBoosting on Decision-Tree models

10-fold cross-validation 10 trials: train on 90%, test on 10% next we report combined accuracy

results

79.883.8

87.089.4 90.1

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

results (full feature-set models)

all significantly better than the prior probability (p<.001)

results (user-centric models)

previous models used information about the buddy (e.g., time since messing that buddy)

can predict different responsiveness for different buddies but what if you wanted just one level of

responsiveness?

built models that did not use any buddy-related features

79.882.5

87.0 89.4 89.3

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

results (user-centric models)

all significantly better than the prior probability (p<.001)

a closer look

(new! not in the paper)

a closer look (new! not in the paper)

analysis of the continuous measure:

log(Time Until Response)

repeated measures ANOVA

Independent Variables: features subset

ParticipantID [Group] as random effect

DF DFDen F p

Group 2 1 0.01 0.992

HourMinutes 1 1257 1.32 0.251

log(ownStat_dur) 1 1564 1.62 0.203

log(timeSincOMsgBdy) 1 1212 7.63 0.006 *

log(timeSincOOther) 1 1416 1.50 0.221

log(buddyStat_dur) 1 1504 0.19 0.667

BaseRelationship 2 1447 1.03 0.357

MessageWindowsCount 1 1462 3.89 0.049 *

FocusedWindowType 18 971 1.47 0.093

FocusedWindowDur 1 1407 0.04 0.840

PrevFocusedWinFeatureDur 1 1567 9.45 0.002 *

MostFocusedWinTime(30) 1 985 0.03 0.865

MostFocusedWinTime(600) 1 957 0.49 0.485

WinSwitchesCountFeature(30) 1 1011 3.99 0.046 *

WinSwitchesCountFeature(600) 1 1089 0.97 0.326

MostFocusedWinType(60) 16 967 1.42 0.122

MostFocusedWinType(300) 20 1026 1.62 0.042 *

MouseEventCountFeature(30) 1 1046 2.98 0.085

MouseDistanceFeature(60) 1 1081 5.08 0.024 *

MouseDistanceFeature(600) 1 1567 1.60 0.206

KBCountFeature(30) 1 996 10.80 0.001 *

KBCountFeature(600) 1 1160 1.99 0.158

Estimate DF DFDen F p

Group 2 1 0.01 0.992

HourMinutes 1 1257 1.32 0.251

log(ownStat_dur) 1 1564 1.62 0.203

log(timeSincOMsgBdy) -0.06852 1 1212 7.63 0.006 *

log(timeSincOOther) 1 1416 1.50 0.221

log(buddyStat_dur) 1 1504 0.19 0.667

BaseRelationship 2 1447 1.03 0.357

MessageWindowsCount 0.08298 1 1462 3.89 0.049 *

FocusedWindowType 18 971 1.47 0.093

FocusedWindowDur 1 1407 0.04 0.840

PrevFocusedWinFeatureDur 0.00001 1 1567 9.45 0.002 *

MostFocusedWinTime(30) 1 985 0.03 0.865

MostFocusedWinTime(600) 1 957 0.49 0.485

WinSwitchesCountFeature(30) -0.16685 1 1011 3.99 0.046 *

WinSwitchesCountFeature(600) 1 1089 0.97 0.326

MostFocusedWinType(60) 16 967 1.42 0.122

MostFocusedWinType(300) Nom 20 1026 1.62 0.042 *

MouseEventCountFeature(30) 1 1046 2.98 0.085

MouseDistanceFeature(60) -0.00001 1 1081 5.08 0.024 *

MouseDistanceFeature(600) 1 1567 1.60 0.206

KBCountFeature(30) -0.00372 1 996 10.80 0.001 *

KBCountFeature(600) 1 1160 1.99 0.158

Estimate DF DFDen F p

Group 2 1 0.01 0.992

HourMinutes 1 1257 1.32 0.251

log(ownStat_dur) 1 1564 1.62 0.203

log(timeSincOMsgBdy) -0.06852 1 1212 7.63 0.006 *

log(timeSincOOther) 1 1416 1.50 0.221

log(buddyStat_dur) 1 1504 0.19 0.667

BaseRelationship 2 1447 1.03 0.357

MessageWindowsCount 0.08298 1 1462 3.89 0.049 *

FocusedWindowType 18 971 1.47 0.093

FocusedWindowDur 1 1407 0.04 0.840

PrevFocusedWinFeatureDur 0.00001 1 1567 9.45 0.002 *

MostFocusedWinTime(30) 1 985 0.03 0.865

MostFocusedWinTime(600) 1 957 0.49 0.485

WinSwitchesCountFeature(30) -0.16685 1 1011 3.99 0.046 *

WinSwitchesCountFeature(600) 1 1089 0.97 0.326

MostFocusedWinType(60) 16 967 1.42 0.122

MostFocusedWinType(300) Nom 20 1026 1.62 0.042 *

MouseEventCountFeature(30) 1 1046 2.98 0.085

MouseDistanceFeature(60) -0.00001 1 1081 5.08 0.024 *

MouseDistanceFeature(600) 1 1567 1.60 0.206

KBCountFeature(30) -0.00372 1 996 10.80 0.001 *

KBCountFeature(600) 1 1160 1.99 0.158

“those in the back can’t see, and those in the front can’t understand…”

Robert Kraut

a closer look (new! not in the paper)

work fragmentation longer time in previous app …. slower more switching (30sec) …. faster longer mouse movements (60sec) …. faster more keyboard activity (30 sec) …. faster more message windows …. slower

longer time since messaging with buddy… faster buddy ID had significant effect

implications for practice

(in the paper)

implications for practice

preserving plausible deniability

making predictions about the receiver, visible to the receiver

multiple concurrent levels of responsiveness

presented statistical models that accurately predict responsiveness to incoming IM based on naturally occurring behavior

we plan to examine using message-content to improve modeling

summary & future work

awareness

message interceptalertmaskenhance

we would like to thank

Mike T (Terry) James Fogarty Darren Gergle Laura Dabbish, and Jennifer Lai

this work was funded in part by NSF Grants IIS-0121560, IIS-0325351, and by DARPA Contract No. NBCHD030010

thank you

for more info visit: www.cs.cmu.edu/~nx6

or email: nx6@cmu.edu

Feature Estimate F p

buddyName[Group,SN] 1.67 0.000 *

log(timeSincOMsgBdy) -0.06852 7.63 0.006 *

PrevFocusedWinFeatureDur 0.00001 9.45 0.002 *

MessageWindowsCount 0.08298 3.89 0.049 *

WinSwitchesCountFeature(30) -0.16685 3.99 0.046 *

MouseDistanceFeature(60) -0.00001 5.08 0.024 *

KBCountFeature(30) -0.00372 10.80 0.001 *

MostFocusedWinType(300) 1.62 0.042 *