Scientific Collaboration& Creativity Lab
Statistical Affect Detection
in Collaborative Chat
CSCW 2013: Mining Social Media Data, Feb. 23
Michael Brooks, Katie Kuksenok, Megan K. Torkildson, Daniel Perry, John J. Robinson, Taylor Jackson Scott, OnaAnicello, Ariana Zukowski, Paul Harris, Cecilia R. Aragon
Scientific Collaboration & Creativity Lab 2/27/2013 2
Scientific Collaboration & Creativity Lab 2/27/2013 3
6:07:57 Ray cool, it worked amusement, relief
6:08:04 Matt woot excitement, joy
6:08:07 Ray awesome, I don't think he needs that long of a sleep after turning it off
acceptance, no affect
6:08:47 We enhanced eready to detect the sticking
no affect
6:08:58 Matt good job supportive, acceptance
6:09:21 seems it did well there happiness, no affect
6:09:26 Ray yeah, pretty cool huh? interest, agreement, happiness
6:09:43 Matt helps keep me from having to stopaicand restart
no affect
6:09:55 Ray indeed, that was the point agreement
June, 2007
Scientific Collaboration & Creativity Lab 2/27/2013 4
• 30 astrophysicists
• US / France
• Daily remote operation of telescope
• Rely on chat to communicate
Nearby Supernova Factory
5
6
Scientific Collaboration & Creativity Lab 2/27/2013 7
SNfactory Chat Logs
• Four years of logs - 449,684 messages
• Manual coding for affective expressions
– 27,344 chat messages coded
– 1-5 coders per message
– 30 affect codes
– Multiple codes allowed
Scott et al. SIGDOC 2012. Adapting Grounded Theory to Construct a Taxonomy of Affect in Collaborative Online Chat.
Scientific Collaboration & Creativity Lab 2/27/2013 8
6:07:57 Ray cool, it worked amusement, relief
6:08:04 Matt woot excitement, joy
6:08:07 Ray awesome, I don't think he needs that long of a sleep after turning it off
acceptance, no affect
6:08:47 We enhanced eready to detect the sticking
no affect
6:08:58 Matt good job supportive, acceptance
6:09:21 seems it did well there happiness, no affect
6:09:26 Ray yeah, pretty cool huh? interest, agreement, happiness
6:09:43 Matt helps keep me from having to stopaicand restart
no affect
6:09:55 Ray indeed, that was the point agreement
June, 2007
Scientific Collaboration & Creativity Lab 2/27/2013 9
Top 13 Affect Codes
4351
3213
1763
1623
1212
1125
975
799
541
518
464
426
369
int…
am…
co…
agr…
an…
co…
acc…
ap…
fru…
sup…
sur…
ant…
ser…
0.808
0.611
0.49
0.491
0.77
0.615
0.657
0.529
0.55
0.583
0.543
0.424
0.602
interest
amusement
considering
agreement
annoyance
confusion
acceptance
apprehension
frustration
supportive
surprise
anticipation
serenity
Times Used Reliability (Kappa)
Scientific Collaboration & Creativity Lab 2/27/2013 10
Linguistic Inquiry and Word Count
(LIWC)
• Detects words for Positive / Negative Emotions
I wish every day could be sunnyand warm. Rain makes me angry.
Positive: 15%Negative: 8%…
Scientific Collaboration & Creativity Lab 2/27/2013 11
11:44:08 Gabri ok that's better relief, serenity
11:44:17 Marcel GREAT ! excitement, happiness, relief, joy
11:44:17 Gabri let's start aic and see anticipation, no affect
11:44:23 Marcel yes ... no affect
11:44:31 Derek Great what? confusion
11:44:32 Gabri can you do that? interest, no affect
11:44:50 derek.. it seems that now the focus is ok no affect
11:45:04 and we can finally start observing no affect
11:45:23 Derek Oh good! happiness, relief, joy
11:45:48 I have been waiting for this moment, because I want to leave the room and get my midnight snack. ;)
amusement
11:46:54 Gabri go... amusement, no affect
11:47:02 and enjoy your snack amusement, no affect
11:47:13 Derek HEhe. amusement
11:47:18 I will bring it back here of course. amusement
June, 2005
Scientific Collaboration & Creativity Lab 2/27/2013 12
The telescope is stuck! >:(
frustration
The telescope is stuuuuuuuuuck...
annoyance
The telescope is stuck??
confusion
Scientific Collaboration & Creativity Lab 2/27/2013 13
• Word counts
• Emoticons
• Word sets– Swear words
– Pronouns
– Negations
– Participant names
• Characters– Capitalization
– Letter repetition
– Punctuation
• Metadata– segment duration, length, rate
Scientific Collaboration & Creativity Lab 2/27/2013 14
• Word counts
• Emoticons
• Word sets– Swear words
– Pronouns
– Negations
– Participant names
• Characters– Capitalization
– Letter repetition
– Punctuation
• Metadata– segment duration, length, rate
Scientific Collaboration & Creativity Lab 2/27/2013 15
Emoticons
Naomi: I think we'd better stopaic... :( sadness
Matt: today was a gym + laundry day :) amusement, happiness
Marcel: and she can't teach over an ssh-channel ;-)
amusement
Scientific Collaboration & Creativity Lab 2/27/2013 16
Word Sets
NegationsPaul: but I wouldn't hazzard a guess apprehension
Ray: cannot talk to camera frustration, no-affect
Swear WordsRay: why the **** doesn't stop_script ******* STOP THE ******* SCRIPT
rage
Matt: ******* ******* ******* I think I broke it frustration, anger, apprehension, embarrassment
Scientific Collaboration & Creativity Lab 2/27/2013 17
Character Features
CapitalizationMarcel: ON TARGET ! relief, joy
Paul: we must set-up adopt an EXPLODING STAR amusement, no-affect
Letter RepetitionRay: noooooooooooooooo, it must be stopped annoyance, anger, fear
Marcel: AAaah too late, they will find meeee amusement
PunctuationRick: looks like something bad happened here... apprehension
Rene: 1 month before max??!? surprise, confusion, considering
Scientific Collaboration & Creativity Lab 2/27/2013 18
Alice: ok, so where was
the ******* SN on the
image?
Feature Value“ok” 1
“telescope” 0
“where” 1
“SN” 1
“image” 1
question marks 1
swears 1
emoticon :) 0
1st person pronouns 0
capitals 2
repetition 0
punctuation 1
length 45
…
Scientific Collaboration & Creativity Lab 2/27/2013 19
Feature importance
Confusion???? length
# question marks"understand"
"confus_""why""what"
"nothing""wrong"
msg. length"thought"
Messages labeled Confusion
Ben: ??? - the answer is likely found in
the otsim code
Marcel: well ... I'm not so sure ...
Gary: Why do we care at all then?
Ray: ummm I mean how does it get to
the header
Scientific Collaboration & Creativity Lab 2/27/2013 20
Feature importance
Apprehension"bad"
"something""problem"
"we""seem""too"
msg. length"not"
# 3rd sg. Pronouns# swearing
Messages labeled Apprehension
Pascal: the problem is than the
automated detection will not work ...
too much galaxy
Ray: But now bad stuff in window
Ben: pascal, we had a problem with
do_fchart
Gabriel: So something is completely
wrong
Scientific Collaboration & Creativity Lab 2/27/2013 21
Feature importance
Amusementemoticon ";)"emoticon ":)"
laughteremoticon ";-)"
"fun"laughter length
"p"# people names
"sleep""of"
Messages labeled Amusement
Kevin: hehe
Ray: hahahaah
Stef: lol ok derek :)
Ray: He never sleeps -- you know that.
Pascal: but I think it could be interesting
for Extreeeeeeeeeeme photometry
study ;-)
Scientific Collaboration & Creativity Lab 2/27/2013 22
Specialized Features
• Count words based on the data
• Medium-specific features
– Emoticons, punctuation…
• Context-specific features
– People names, jargon…
• Affect-specific features
– Swearing vs. emoticons
Scientific Collaboration & Creativity Lab 2/27/2013 23
5:17:48 Marcel ok, so let's cycle the stuff
5:18:04 Rick ok…
5:18:40 Marcel damn mouse cutandpast
5:19:03 Ray off 1 right? then on 1?
5:19:32 Marcel have you telnet sdsugreen ??
5:19:58 Ray director on lbl2 looks dead
5:20:34 Marcel ok, one thind at a time. have you cycled the baytech on sdsugreen ?
5:20:36 Ray what is best way to revive it
5:20:39 baytech
5:20:40 yes
5:20:46 not sdsu
5:21:08 go ahead and do it I am not evneon this **** shift...grrr
5:21:22 Marcel ok, maybe we have to kill director and restart it mkanually
5:21:32 Ray yeah but that's tricky; all these damn arguments
5:23:53 Rick emile, I have no idea what's going on here
5:23:57 only that it is bad
September, 2006
Scientific Collaboration & Creativity Lab 2/27/2013 24
5:17:48 Marcel ok, so let's cycle the stuff
5:18:04 Rick ok…
5:18:40 Marcel damn mouse cutandpast
5:19:03 Ray off 1 right? then on 1?
5:19:32 Marcel have you telnet sdsugreen ??
5:19:58 Ray director on lbl2 looks dead
5:20:34 Marcel ok, one thind at a time. have you cycled the baytech on sdsugreen ?
5:20:36 Ray what is best way to revive it
5:20:39 baytech
5:20:40 yes
5:20:46 not sdsu
5:21:08 go ahead and do it I am not evneon this **** shift...grrr
5:21:22 Marcel ok, maybe we have to kill director and restart it mkanually
5:21:32 Ray yeah but that's tricky; all these damn arguments
5:23:53 Rick emile, I have no idea what's going on here
5:23:57 only that it is bad
September, 2006
Scientific Collaboration & Creativity Lab 2/27/2013 25
Classifier F-measure Precision Recall Accuracy
Naïve Bayes 0.650 0.637 0.691 0.637
Logistic Reg. 0.730 0.731 0.731 0.730
SVM (SMO) 0.759 0.766 0.751 0.761
C4.5 (J48) 0.700 0.724 0.680 0.710
Scientific Collaboration & Creativity Lab 2/27/2013 26
Support Vector Machine
• Accurate
• Fast
• Transparent
# swear words
“frustration” applies“frustration” does not apply
# “
ok”
Scientific Collaboration & Creativity Lab 2/27/2013 27
Support Vector Machine
• Accurate
• Fast
• Transparent
# swear words
?
“frustration” applies“frustration” does not apply
# “
ok”
Scientific Collaboration & Creativity Lab 2/27/2013 28
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
interest
amusement
considering
agreement
annoyance
confusion
acceptance
apprehension
frustration
supportive
surprise
anticipation
serenity
Precision Recall
Scientific Collaboration & Creativity Lab 2/27/2013 29
Interpretability
• How is the classifier making decisions?
• What features are important in the model?
# swear words# “
ok”
“frustration” applies“frustration” does not apply
Scientific Collaboration & Creativity Lab 2/27/2013 30
Feature importance
Amusementemoticon ";)"emoticon ":)"
laughteremoticon ";-)"
"fun"laughter length
"p"# people names
"sleep""of"
Messages labeled Amusement
Kevin: hehe
Ray: hahahaah
Stef: lol ok derek :)
Ray: He never sleeps -- you know that.
Pascal: but I think it could be interesting
for Extreeeeeeeeeeme photometry
study ;-)
Scientific Collaboration & Creativity Lab 2/27/2013 31
Interpretable Classifiers
• Explain classification errors
• Suggest improvement strategies
• Discover interesting anomalies
Scientific Collaboration & Creativity Lab 2/27/2013 32
Future Work
Scientific Collaboration & Creativity Lab 2/27/2013 33
Sequential Modeling
5:19:58 Ray director on lbl2 looks dead
5:20:34 Marcel ok, one thind at a time. have you cycled the baytech on sdsugreen ?
5:20:36 Ray what is best way to revive it
5:20:39 baytech
5:20:40 yes
5:20:46 not sdsu
5:21:08 go ahead and do it I am not evneon this **** shift...grrr
5:21:22 Marcel ok, maybe we have to kill director and restart it mkanually
5:21:32 Ray yeah but that's tricky; all these damn arguments
5:23:53 Rick emile, I have no idea what's going on here
5:23:57 only that it is bad
Scientific Collaboration & Creativity Lab 2/27/2013 34
Interactive Visual Analysis
Scientific Collaboration & Creativity Lab 2/27/2013 35
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
Nu
mb
er
of
Tw
eets
Time (EST), 2/3/2013positive negative neutral
Affect in Twitter
kick
off
gam
e o
ver
gam
e re
sum
es
bla
cko
ut
gam
e re
sum
es
hal
ftim
e
Scientific Collaboration & Creativity Lab 2/27/2013 36
Classify…• Positive/negative/neutral
sentiment
• Highly granular emotions
• Anything else you can label
In…• longer, formal documents (blog
posts, reviews)
• individual sentences
• instant messages
• tweets
• Anything else you can put in CSV
github.com/etcgroup/aloe
Download it, use it, & tell us what you think!
Michael [email protected]
http://depts.washington.edu/sccl
Top Related