![[IRT] Item Response Theory Reference Manual](https://static.fdocuments.us/doc/165x107/5879ee1c1a28ab82408bc4f0/irt-item-response-theory-reference-manual.jpg)
Which Work-Item Updates Need Your Response?
13
Why is Tim wearing such a jaded look this morning ? What is he staring at ? 1
-
Upload
debdoot-mukherjee -
Category
Technology
-
view
395 -
download
0
description
Work-item notifications alert the team collaborating on a work-item about any update to the work-item (e.g., addition of comments, change in status). However, as software professionals get involved with multiple tasks in project(s), they are inundated by too many notifications from the work-item tool. Users are upset that they often miss the notifications that solicit their response in the crowd of mostly useless ones. We investigate the severity of this problem by studying the work-item repositories of two large collaborative projects and conducting a user study with one of the project teams. We find that, on an average, only 1 out of every 5 notifications that are received by the users require a response from them. We propose TWINY -- a machine learning based approach to predict whether a notification will prompt any action from its recipient. Such a prediction can help to suitably mark up notifications and to decide whether a notification needs to be sent out immediately or be bundled in a message digest. We conduct empirical studies to evaluate the efficacy of different classification techniques in this setting. We find that incremental learning algorithms are ideally suited, and ensemble methods appear to give the best results in terms of prediction accuracy.
Transcript of Which Work-Item Updates Need Your Response?
Why is Tim wearing such a jaded look this
morning ?
What is he staring
at ?1
Deluge of Notifications for
Work-Item Updates !!
Deluge of Notifications for
Work-Item Updates !!
Mostly Useless
2
Which Work-Item Updates Need Your
Response?The 10th Working Conference on Mining Software Repositories
Which Work-Item Updates Need Your
Response?The 10th Working Conference on Mining Software Repositories
Debdoot Mukherjee, IBM Re s e a rch - Ind iaMalika Garg, Ind ia n Ins titute o f Te chno lo g y - De lhi
Why this problem
?
Why this problem
?How
severe ?How
severe ?Solution
IdeaSolution
IdeaHow
effective ?How
effective ?Talk Outline :
3
Only a fraction of the subscribedwork-items are of primary concern. Input is only sparingly necessary in the rest
10 Working Spheres
People receive MANY MORE notifications compared to the number of their own updates …
1 Out of 5 Notifications
require a response
Leads more severely impacted
than Individual
Contributors
4
Project Manager “Up da te s fro m m y c lie nt re q uire urg e nt re s p o ns e s . . . … . re s t a re m o s tly FYI ite m s . ”
Which notifications are perceived as important?
Developer“Us ua lly , I o nly re a d no tific a tio ns fro m the wo rk-ite m s tha t I o wn. Fo r o the rs , I ne e d to p a y a tte ntio n o nly if m y na m e is m e ntio ne d . ”
Architect“I a m re s p o ns ible fo r re v ie wing c o d e c o m m its m a d e by the te a m , s o I ke e p a n e y e o n up da te s tha t ha ve linke d cha ng e -s e ts .”
5
6
Predict whether a user shall respond to a notification by
learning from users’ responses to similar past
notifications
Notifications that do not demand any response but are important since they improve awareness are NOT in
scope.7
8
Creating Notification Examples
9
Labeling Notification Examples
Label a notification example generated for a user as Re s p o ns e -Re q d , if (s)he updates the work-item s o o n after receiving the notification and before others update it. Otherwise, the example is labeled as No -Re s p o ns e -Re q d .
10
Model Training Considerations
11
Efficacy of different types of classifiers Naïve Bayes, Decision Trees, Support Vector Machines
Notifications keep streaming in – which window to use as training set ? Avert this question by use of incremental classifiers
Distribution of many features may change drastically with the churn in software projects Use of adaptive classifiers to combat project dynamics Ensemble classifiers to deal with evolving data
Empirical Evaluation: Summary
12
13
