Proceedings of NAACL-HLT 2018: Student Research Workshop, pages 136–143New Orleans, Louisiana, June 2 - 4, 2018. c©2017 Association for Computational Linguistics

Sensing and Learning Human AnnotatorsEngaged in Narrative Sensemaking

McKenna K. Tornblad,1 Luke Lapresi,2 Christopher M. Homan,2Raymond W. Ptucha3 and Cecilia Ovesdotter Alm4

1College of Behavioral and Health Sciences, George Fox University2Golisano College of Computing and Information Sciences, Rochester Institute of Technology

3Kate Gleason College of Engineering, Rochester Institute of Technology4College of Liberal Arts, Rochester Institute of Technology

mtornblad14@georgefox.edu, lxl6996@rit.edu, cmh@cs.rit.edu,rwpeec@rit.edu, coagla@rit.edu

Abstract

While labor issues and quality assurance incrowdwork are increasingly studied, how an-notators make sense of texts and how they arepersonally impacted by doing so are not. Westudy these questions via a narrative-sortingannotation task, where carefully selected (bysequentiality, topic, emotional content, andlength) collections of tweets serve as exam-ples of everyday storytelling. As readers pro-cess these narratives, we measure their facialexpressions, galvanic skin response, and self-reported reactions. From the perspective of an-notator well-being, a reassuring outcome wasthat the sorting task did not cause a mea-surable stress response, however readers re-acted to humor. In terms of sensemaking,readers were more confident when sorting se-quential, target-topical, and highly emotionaltweets. As crowdsourcing becomes more com-mon, this research sheds light onto the percep-tive capabilities and emotional impact of hu-man readers.

1 Introduction

A substantial sector of the gig economy is theuse of crowdworkers to annotate data for machinelearning and analysis. For instance, storytellingis an essential human activity, especially for in-formation sharing (Bluvshtein et al., 2015), mak-ing it the subject of many data annotation tasks.Microblogging sites such as Twitter have reshapedthe narrative format, especially through characterrestrictions and nonstandard language, elementswhich contribute to a relatively unexplored modeof narrative construction; yet, little is known aboutreader responses to such narrative content.

We explore reader reactions to narrative sense-making using a new sorting task that varies thepresumed cognitive complexity of the task and

elicits readers’ interpretations of a target topic andits emotional tone. We carefully and systemati-cally extracted 60 sets of tweets from a 1M-tweetdataset and presented them to participants via aninterface that mimicks the appearance of Twitter(Figure 1). We asked subjects to sort chronolog-ically the tweets in each set, half with true narra-tive sequence and half without. Each set consistedof 3–4 tweets from a previously collected corpus,where each tweet was labeled using a frameworkby Liu et al. (2016) as work-related or not. Inaddition to sequentiality and job-relatedness, setswere evenly distributed across two other variableswith two levels each, of interest for understand-ing narrative sensemaking (Table 1). We recordedreaders’ spoken responses to four questions (Fig-ure 2) about each set, which involved the sort-ing task, reader confidence, topical content (job-relatedness), and emotional tone. We used gal-vanic skin response (GSR) and facial expressionanalysis to explore potentially quantifiable met-rics for stress-based reactions and other aspects ofreader-annotator response. Our results add under-standing of how annotators react to and processeveryday microblog narratives.

This study makes these contributions:1) Opens a dialogue on annotator well-being;2) Presents a method to study annotator reactions;3) Indicates that narrative sorting (task with de-grees of complexity) does not cause an increasedstress response as task complexity increases;4) Studies the role of topic saliency and emotionaltone in narrative sense-making; and5) Probes how features model annotator reactions.

2 Related Work

That annotation tasks cause fatigue is widely rec-ognized (Medero et al., 2006), and crowdsourcing

136

Figure 1: The experimental setup showing the experi-menter’s screen, a view of the participant on a secondmonitor, and a closeup view of the participant’s screen.A fictional tweet set is shown to maintain privacy.

1. What is the correct chronological order ofthese tweets?

2. How confident are you that your sorted se-quence is correct?

3. Are these tweets about work? [Target topic]If so, are they about starting or leaving a job?

4. What is the dominant emotion conveyed inthis tweet set?

Figure 2: The four study questions that participants an-swered for each tweet set in both trials.

annotations has been critiqued for labor market is-sues, etc. (Fort et al., 2011). Another importantconcern is how stressful or cognitively demandingannotation tasks may adversely impact annotatorsthemselves, a subject matter that has not yet beenprominently discussed, despite its obvious ethicalimplications for the NLP community and machinelearning research.

Microblogging sites raise unique issues aroundsharing information and interacting socially.Compared to traditional published writing, Twitterusers must choose language that conveys meaningwhile adhering to brevity requirements (Barnard,2016). This leads to new narrative practices, suchas more abbreviations and nonstandard phraseol-ogy, which convey meaning and emotion differ-ently (Mohammad, 2012). Tweets are also pub-lished in real time and the majority of users maketheir tweets publicly available, even when theirsubject matter is personal (Marwick and Boyd,

2011). Stories are shared among—and everydaynarrative topics are explored by—communities,providing a window into the health and well-beingof both online and offline networks at communitylevels.

We selected job-relatedness as our Target Topicvariable. Employment plays a prominent rolein the daily lives and well-being of adults andis a popular theme in the stories microbloggersshare of their lives. One study reported thatslightly more than a third of a sample of nearly40,000 tweets by employed people were work-related (van Zoonen et al., 2015). Employees canuse social media to communicate with coworkersor independently to share their experiences (Mad-sen, 2016; van Zoonen et al., 2014), providingmany interesting job-related narratives. Because itis so common in on- and off-line storytelling, an-notators can relate to narratives about work. Workis also a less stressful topic, compared to otherones (Calderwood et al., 2017). Tweets with highor low emotional content in general may also af-fect readers in ways that change how they under-stand texts (Bohn-Gettler and Rapp, 2011). Weused the sentiment functionality of TextBlob1 todistinguish high- and low-emotion tweets in thestudy.

Variable Lvl 1 Abbr. Lvl 2 Abbr.

Narrative Seq. yes seq+ no seq-Target Topic yes job+ no job-

Emo. Content high emo+ low emo-Num. Tweets 3 4

Table 1: Overview of the four study variables that char-acterize each tweet set. Lvl 1 indicates the first condi-tion for a variable, and Lvl 2 indicates the second. Theprimary distinction is between sets in Trial 1 (all seq+)and 2 (all seq-).

3 Study Design

The study involved 60 total tweet sets across twotrials, reflecting a presumed increase in the cog-nitive complexity of the narrative sorting task.Trial 1 consisted of sets with a true narrative or-der (seq+) and Trial 2 of sets without (seq-); withthe assumption that the latter is more cognitivelydemanding. The tweets were evenly distributedacross the other three study variables (Table 1).

1https://textblob.readthedocs.io

137

For selecting job+ sets in Trial 1, queries ofthe corpus combined with keywords (such ascoworker, interview, fired and other employment-related terms) identified job-related seed tweetswhile additional querying expanded the set for thesame narrator. For example, a 3-item narrativecould involve interviewing for a job, being hired,and then starting the job. For the job- sets in Trial1, queries for seed tweets focused on keywords forother life events that had the potential to containnarratives (such as birthday, driver’s license, andchild), continuing with the same method used forjob+ sets. For each Trial 2 set, we conducted simi-lar keyword queries, except that we chose the seedtweet without regard to its role in any larger nar-rative. We selected the rest of the tweets in theset to match and be congruent with the same userand job+/job- and emo+/emo- classes as the seedtweet. The final selection of 60 tweet sets wasbased on careful manual inspection.

4 Methods

Participants: Participants were nineteen individ-uals (21% women) in the Northeastern U.S. whoranged in age from 18 to 49 years (M = 25.3).58% were native English speakers, and activeTwitter users made up 42% of the sample.Measures: Galvanic Skin Response (GSR): Weused a Shimmer3 GSR sensor with a sampling rateof 128 Hz to measure participants’ skin conduc-tance in microsiemens (µS). More cognitively dif-ficult tasks may induce more sweating, and higherµS, corresponding to a decrease in resistance andindicating cognitive load (Shi et al., 2007). Weused GSR peaks, as measured by iMotions soft-ware (iMotions, 2016), to estimate periods of in-creased cognitive arousal.

Facial Expression: To also differentiate be-tween positive and negative emotional arousal,we captured and analyzed readers’ expressed fa-cial emotions while interacting with the task usingAffectiva’s facial expression analysis in iMotions(McDuff et al., 2016; iMotions, 2016).Procedure: Participants sat in front of a monitorwith a Logitech C920 HD Pro webcam and werefitted with a Shimmer3 GSR sensor worn on theirnon-dominant hand. They then completed a de-mographic pre-survey while the webcam and GSRsensor were calibrated within the iMotions soft-ware. Next, participants completed a practice trialwith an unsorted tweet set on the left-hand side of

the screen and questions on the right (Figure 1).Participants answered each question aloud,

minimizing movement, and continued to the nextquestion by pressing a key. We provided a visualof Plutchik’s wheel of emotions (Plutchik, 2001) ifthe participant needed assistance in deciding on anemotion for Question 4 (Figure 2), although theywere free to say any emotion word. After the prac-tice trial, the participant completed Trial 1 (seq+),followed by a short break, then Trial 2 (seq-). Fi-nally, we gave each participant a post-study sur-vey on their self-reported levels of comfort withthe equipment and of cognitive fatigue during eachtrial.

One experimenter was seated behind the partic-ipant in the experiment room with a separate videoscreen to monitor the sensor data and answer anyquestions. A second experimenter was in anotherroom recording the participants’ answers as theywere spoken via remote voice and screen sharing.Classifier: Pairing each tweet set with each partic-ipant’s response data as a single item yields 1140data items (19 subjects * 60 sets). We used theLIBSVM (Chang and Lin, 2011) support vectormachine (SVM) classifier library, with a radial ba-sis function kernel and 10-fold cross-validation, topredict variables of interest in narrative sensemak-ing. Table 2 lists observed and predicted features.When a feature served as a prediction target it wasnot used as an observed one. The mean valueof the feature was used in cases of missing datapoints.

Study Self-Report Sensing

Narrative Seq. Kendall τ Dist. Facial ExpressionEmo. Content Confidence Average GSRTarget Topic Topic Judgment Num. GSR PeaksNum. Tweets . Dom. Emotion Time

Twitter UserSoc. Media Use

Table 2: Features used for a SVM classifier were putinto three groups: 1) Study variables; 2) Subject self-reported measures from participants’ answers to studyquestions and pre-survey responses related to socialmedia use; and 3) Sensing attributes relating to col-lected sensor and time data.

5 Results and Discussion

Trial Question 1: Tweet Sorting To quantify howclose participants’ sorted orders were to the cor-

138

rect narrative sequence for each tweet set in Trial1, we used the Kendall τ rank distance betweenthe two, or the total number of pairwise swaps ofadjacent items needed to transform one sequenceinto the other. An ANOVA revealed that partici-pants were significantly closer to the correct nar-rative order when tweets were job-related (job+),compared to not job-related (job-), regardless ofemotional content, F (1, 566) = 13.30, p < .001(Figure 3a). This result indicates that the targettopic was useful for temporally organizing partsof stories. Without this topic’s frame to start from,readers were less accurate in the sorting task.

Figure 3: The four panels show: (a) Average Kendallτ rank distance between participants’ orders and thecorrect order by job-relatedness (target topic) and emo-tional content in Trial 1 only. Participants were signif-icantly more accurate with sets about the target topic,regardless of emotional content. (b) Average Kendallτ rank distance between participants’ sorted orders andthe correct order by confidence level in Trial 1. 0 =No Confidence and 3 = High Confidence. Participantstended to be closer to the correct order as their confi-dence increased. (c) Average confidence ratings acrossboth trials by job-relatedness and emotional content.The only non-significant difference was between Job-Related (job+) x High Emotional Content (emo+), andJob-Related (job+) x Low Emotional Content (emo-)groups (see Table 3), indicating an interaction betweentarget topic and emotional content. (d) Average timespent (ms) per tweet set by confidence and trial. Par-ticipants spent overall less time on sets in Trial 2 thanTrial 1, and less time for sets in both trials as confidenceincreased.

Trial Question 2: Confidence An ANOVAshowed that as participants became more confident

Grp 1 Grp 2 t p SE d

Y × H Y × L 1.646 .101 0.066 0.097Y × H N × H 4.378 <.001* 0.062 0.259Y × H N × L 10.040 <.001* 0.063 0.595Y × L N × H 2.541 .012† 0.064 0.151Y × L N × L 8.030 <.001* 0.065 0.476N × H N × L 5.750 <.001* 0.063 0.341

Table 3: Student’s t-test indicates differences in con-fidence by job-relatedness and emotional content (vi-sualized in Figure 3c) with df=284 for all conditions.Groups are named with the following convention: Job-Relatedness x Emotional Content. Y = Yes (job+), N= No (job-), H = High (emo+), and L = Low (emo-).(* p < .001 † p < .05).

in Trial 1, their sorted tweet orders were closer tothe correct order, F (3, 566) = 14.72, p < .001.This demonstrated that readers are able to accu-rately estimate their correctness in the sorting taskwhen tweets had a true narrative sequence (Fig-ure 3b). This is an interesting result suggestingthat participants were not under- or overconfidentbut self-aware of their ability to complete the sort-ing task. An ANOVA also showed that partici-pants were significantly more confident in Trial1 (M = 2.20, SD = 0.79) than Trial 2 (M =1.50, SD = 0.92), F (1, 1138) = 188.00, p <.001, regardless of other study variables (Fig-ure 4). This indicates that it was more difficultfor participants to assign a sorted order when thetweets did not have one.

Figure 4: Total instances of each confidence level bytrial (0 = No Confidence and 3 = High Confidence).

Because each participant’s confidence scoresfor tweet sets in Trial 1 were not related to scoresin Trial 2, we used an independent samples t-testand found that confidence was significantly higherfor job+ tweets (M = 2.05, SD = 0.84) than

139

job- tweets (M = 1.65, SD = 0.97), t(1138) =−7.38, p < .001. By including emotional con-tent in an ANOVA, we also found that partic-ipants were significantly more confident aboutemo+ tweets compared to emo-, but only whenthe topic was not job-related (job-), F (1, 1136) =5.65, p = .018. Comparisons among groups thesegroups can be seen in Table 3 and Figure 3c.

These findings suggest that the target topic, hav-ing a known narrative frame, was the most usefulpiece of information for participants in orderingtweets. When there was no job narrative to guideinterpretation, emotional context was instead usedas the primary sorting cue. This result agrees withprevious work (Liu et al., 2016) by suggesting thatthe job life cycle involves a well-known narrativesequence that is used as a reference point for sort-ing tweets chronologically.

Confidence level could indicate cognitive load,which is supported by the ANOVA result thatparticipants spent less time for each tweet setas their confidence increased, regardless of trial,F (3, 1103) = 16.71, p < .001 (Figure 3d).This suggests that participants spent less time onsets that appeared to be easier to sort and moretime on sets that were perceived as more diffi-cult. This could be a promising factor to predicthow straightforward sorting different texts will bebased on readers’ confidence and time spent.

However, participants also spent significantlymore time for each tweet set in Trial 1 than Trial2 regardless of confidence level, F (1, 1103) =29.53, p < .001 (Figure 3d), even though Trial 2was expected to be more difficult and thus time-consuming. This contrary result could be for sev-eral reasons. First, participants may have simplygot faster through practice. Second, they may havebeen fatigued from the task and sped up to finish.Lastly, participants could have given up on the taskmore quickly because it was more difficult, an in-dicator of cognitive fatigue.

Trial Question 3: Target Topic Participants cor-rectly inferred the topic as job-related 97.5% of thetime (true positive rate) and not job-related 96.3%(true negative rate) of the time. For more ambigu-ous sets, we observed that participants added qual-ifiers to their answers more often, such as “thesetweets could be about starting a job,” but were stillable to determine the topic. These observations in-dicate that readers perform well when determiningthe topical context of tweets despite the format pe-

culiarities that accompany the text.

Trial Question 4: Dominant Emotion If partic-ipants gave more than one word to answer thisquestion, we used only the first to capture theparticipant’s initial reaction to the text. We cate-gorized words according to Plutchick’s wheel ofemotions (2001), and used the fundamental emo-tion in each section of the wheel (such as joy andsadness) as the category label. Emotion words inbetween each section were also used as labels anda neutral category was added, resulting in 17 cate-gories explored in classification (below).

Sensor Data Analysis We averaged GSR read-ings across all four study questions for each tweetset. Because participants spent minutes answeringquestions 1 and 4 compared to mere seconds on 2and 3, we chose to examine overall readings foreach set instead of individual questions. An ex-tremely short timeframe yields fewer data pointsand more impact in the event of poor measure-ment. We were also more interested in differencesbetween the study variables of the tweet sets (Ta-ble 1) rather than differences by question, althoughthis could be a direction for future data analysisand research.

Interestingly, an ANOVA indicated no differ-ences in overall normalized GSR levels or num-ber of peaks across any of the four study variables,which leads to important insights. First, it suggeststhat annotators may not feel more stressed whentrying to sort texts that have no true narrative se-quence. This could be because piecing togethernarratives is so fundamental and natural in humaninteraction and cognition. Second, when the focusis the sorting task, it appears that engaging withtweet sets with high emotional content—whetherabout or not about the target topic—did not elicit agreater stress response in readers. This adds moreunderstanding to previous research (Calderwoodet al., 2017) on emotional and physiological re-sponses to texts.

We analyzed facial expressions using Affec-tiva (McDuff et al., 2016), obtaining the total num-ber of instances per minute of each facial expres-sion (anger, contempt, disgust, fear, joy, sadness,and surprise) for each tweet set by participant. Werecorded the emotion that was displayed most of-ten by a participant for a given tweet set (neutralif no emotion registered over Affectiva’s thresh-old of 50). Disgust and contempt appeared to bethe primary emotions displayed across all narra-

140

tives regardless of topic or emotional content.Weobserved that participants’ neutral resting facestended to be falsely registered as disgust and con-tempt. Other factors such as skin tone and glassesinfluenced facial marker tracking.

By observation it was clear that participants hadreactions (Figure 5) to tweet sets that were funny,including complex forms of humor such as ironyand sarcasm. This is an important finding becauseannotating humor is a task that human readers arevery good at compared to machines. The facial re-sponse to humor when reading texts could be usedin a machine learning-based model to identify andclassify humorous content.

Figure 5: Participant displaying a neutral facial expres-sion followed by a joy reaction to a humorous tweet.

Classification Results To further study the col-lected human data and explore patterns of inter-est in the data, we used SVMs to model a tweetset’s Emotional Content (emo+/-) and Narra-tive Sequence (seq+/-); and a participant’s Confi-dence and self-reported Dominant Emotion (Dom.Emo.) (see Table 4). The label set differed foreach classification problem, ranging from binary(emo+/- and seq+/-) to multiple, less balancedclasses (4 for confidence and 17 for dominantemotion). When the label being predicted was alsopart of the observed feature group used to build theclassifier, it was excluded from the feature set.

Table 4 displays the accuracy for these classi-fiers, using various combinations of the featuressets from Table 2. It also shows, for each variablemodeled, results from a Good-3 selection of fea-tures. This approach uses top-performing featuresvia exhaustive search over all valid combinationsof three. The same Good-3 features (Table 5) wereused for all trials for a classification problem in Ta-ble 4.

Often, either All or Good-3 sets result in higherperformance. Confidence and emo+/- classifica-tion improves performance with Trial 1 classifica-tion; however, since the dataset is modest in size,

Feat. Conf. Emo+/- Dom. Emo. Seq+/-

T1 T2 C T1 T2 C T1 T2 C C

Leave 1-subject out cross-validation

Study 49 41 45 53 53 53 32 34 33 53Rep. 48 35 43 65 62 59 28 28 26 67Sens. 43 39 37 54 56 52 26 22 25 49All 46 38 42 62 64 62 35 32 34 68

Good-3 49 42 45 71 62 66 34 30 32 70Leave 1-question out cross-validation

Study 46 41 44 27 27 53 23 19 21 53Rep. 47 40 42 63 58 43 21 22 20 64Sens. 45 43 39 45 52 44 24 21 25 51All 47 41 45 39 44 42 26 19 23 63

Good-3 50 41 46 55 31 57 34 25 30 67Leave 1-subject-and-question out cross-validation

Study 49 45 47 27 27 27 35 36 34 27Rep. 46 41 43 64 57 56 24 27 25 68Sens. 46 42 39 52 55 50 26 24 25 59All 50 42 47 57 61 60 35 31 34 74

Good-3 51 42 46 71 62 66 34 33 33 70

Table 4: Classification accuracies rounded to nearestpercent for Trial 1 (T1), Trial 2 (T2) and both trialscombined (C). Bold values indicate the most accuratefeature set’s prediction percentage per trial or com-bined. Because trials 1 and 2 differed in having a truenarrative sequence, no seq+/- prediction is reported bytrial.

it is difficult to make a judgment as to whether ornot presenting users with chronologically orderedtweets yield better classifiers for narrative content.As expected, regardless of which set of features isbeing used, simpler boolean problems outperformthe more difficult multiclass ones.

6 Conclusion and Future Work

This study adds understanding of how annotatorsmake sense of microblog narratives and on the im-portance of considering how readers may be im-pacted by engaging with text annotation tasks.

The narrative sorting task—and the self-evaluated confidence rating—appears useful forunderstanding how a reader may frame and inter-pret a microblog narrative. Confidence displayed astrong relationship with several factors, includingtarget topic, time spent, Kendall τ distance, andcognitive complexity between trials. This pointsto the importance of considering confidence in an-

141

Confidence Emo+/- Dom. Emo. Seq+/-

Seq+/- # of Tweets # of Tweets Job+/-Kendall τ dist. Job+/- Emo+/- K. τ dist.Soc. Med. use Dom. Emo. Seq+/- Confid.

Table 5: Good-3 features used for each SVM classifier.

notation. Confidence ratings can also help iden-tify outlier narratives that are more challenging toprocess and interpret. The increase in cognitivecomplexity in Trial 2 did not appear to cause a po-tentially unhealthy stress response in annotators.

Despite generating interesting results, this studyhad limitations. For example, the sample size wasmodest and trial order was not randomized. Ad-ditionally, the topics of tweets were not overlystressful, and we avoided including tweets wethought could trigger discomfort. As an ex-ploratory study, the quantitative results presentedrepresent preliminary findings. More nuanced andadvanced statistical analysis is left for future work.

Future work could benefit from developing clas-sifiers for predicting whether a microblog post ispart of a narrative; a useful filtering task completedby careful manual inspection in this study. Ad-ditional development of classifiers will focus onfurther aspects related to how readers are likely tointerpret and annotate microblog narratives.

Acknowledgments

This material is based upon work supported bythe National Science Foundation under Award No.IIS-1559889. Any opinions, findings, and conclu-sions or recommendations expressed in this ma-terial are those of the author(s) and do not nec-essarily reflect the views of the National ScienceFoundation.

ReferencesJosie Barnard. 2016. Tweets as microfiction: On Twit-

ter’s live nature and 140-character limit as tools fordeveloping storytelling skills. New Writing: The In-ternational Journal for the Practice and Theory ofCreative Writing, 13(1):3–16.

Marina Bluvshtein, Melody Kruzic, and Victor Mas-saglia. 2015. From netthinking to networking to net-feeling: Using social media to help people in jobtransitions. The Journal of Individual Psychology,71(2):143–154.

Catherine M Bohn-Gettler and David N Rapp. 2011.Depending on my mood: Mood-driven influences

on text comprehension. Journal of Educational Psy-chology, 103(3):562.

Alexander Calderwood, Elizabeth A Pruett, RaymondPtucha, Christopher Homan, and Cecilia O Alm.2017. Understanding the semantics of narrativesof interpersonal violence through reader annota-tions and physiological reactions. In Proceedings ofthe Workshop on Computational Semantics beyondEvents and Roles, pages 1–9.

Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACMTransactions on Intelligent Systems and Technology,2:27:1–27:27.

Karen Fort, Gilles Adda, and K Bretonnel Cohen.2011. Amazon Mechanical Turk: Gold mine or coalmine? Computational Linguistics, 37(2):413–420.

iMotions. 2016. iMotions Biometric Research Plat-form 6.0.

Tong Liu, Christopher Homan, Cecilia OvesdotterAlm, Megan C Lytle, Ann Marie White, andHenry A Kautz. 2016. Understanding discourse onwork and job-related well-being in public social me-dia. In Proceedings of the 54th Annual Meetingof the Association for Computational Linguistics,pages 1044–1053.

Vibeke Thøis Madsen. 2016. Constructing organi-zational identity on internal social media: A casestudy of coworker communication in Jyske Bank.International Journal of Business Communication,53(2):200–223.

Alice E Marwick and Danah Boyd. 2011. I tweet hon-estly, I tweet passionately: Twitter users, contextcollapse, and the imagined audience. New Media& Society, 13(1):114–133.

Daniel McDuff, Abdelrahman Mahmoud, Moham-mad Mavadati, May Amr, Jay Turcot, and Rana elKaliouby. 2016. AFFDEX SDK: A cross-platformreal-time multi-face expression recognition toolkit.In Proceedings of the 2016 CHI Conference Ex-tended Abstracts on Human Factors in ComputingSystems, CHI EA ’16, pages 3723–3726, New York,NY, USA. ACM.

Julie Medero, Kazuaki Maeda, Stephanie Strassel, andChristopher Walker. 2006. An efficient approach togold-standard annotation: Decision points for com-plex tasks. In Proceedings of the Fifth InternationalConference on Language Resources and Evaluation,volume 6, pages 2463–2466.

Saif M Mohammad. 2012. #Emotional tweets. In Pro-ceedings of the First Joint Conference on Lexicaland Computational Semantics, pages 246–255. As-sociation for Computational Linguistics.

Robert Plutchik. 2001. The Nature of Emotions.American Scientist, 89:344.

142

Yu Shi, Natalie Ruiz, Ronnie Taib, Eric Choi, and FangChen. 2007. Galvanic skin response (GSR) as an in-dex of cognitive load. In Proceedings of CHI ’07Extended Abstracts on Human Factors in Comput-ing Systems, CHI EA ’07, pages 2651–2656, NewYork, NY, USA. ACM.

Ward van Zoonen, Toni GLA van der Meer, andJoost WM Verhoeven. 2014. Employees work-related social-media use: His master’s voice. PublicRelations Review, 40(5):850–852.

Ward van Zoonen, Joost WM Verhoeven, and RensVliegenthart. 2015. How employees use Twitter totalk about work: A typology of work-related tweets.Computers in Human Behavior, 55:329–339.

143