Wearable Apparatus For Quantifying Social-Emotional ... · Wearable Apparatus For Quantifying...
1
Wearable Apparatus For Quantifying Social-Emotional Expression Recognition in Natural Face-to-Face Interactions Alea Teeters, Rana El Kaliouby, Matthew Goodwin, Mia Shandell, Rosalind W. Picard Media Laboratory, Massachusetts Institute of Technology & The Groden Center Objectives There is a need for tools that measure social-emotional expression recognition in face-to-face conversations to establish an accurate baseline of recognition ability and to evaluate the efficacy of social interventions. Methods We set up dyadic conversations ; each participant wore Self-Cam to record facial and head movements. >>Data Collection Autistic-Typical pairs: 6 male students at the Groden Center with a diagnosis of mild to moderate autism spectrum disorder (ADOS) with anxiety disorder, ADHD and down syndrome interact with a staff member twice a week for 10 minutes over 2 months. Typical-Typical pairs: 4 volunteers (2males, 2 females) from the MIT Media Lab were split into two pairs and asked to carry on a conversation for at least10 minutes. >>Segmenting and Labeling the Videos From a total of 100 videos, 2 segmenters chose clips that show: disagreeing, thinking, unsure, confused, inter- ested, uninterested, angry, smiling, happy, sad, surprised, concerned, distracted. Then, 10 independent labelers (who scored above 28/36 on Baron-Cohen’s Eyes Test) view each video and pick zero or more of the above labels. For 5 (out of 13) labels--thinking, interested, unsure, happy, and smiling--we chose the clips with the higest inter-rater reliability. These 180 clips form the test questions. Results The test was taken by 3 groups: 4 raters, 4 controls, and 5 out of the 6 Groden students (subject 2 was no longer available). Conclusions Our wearable self-cam technology successfully gathered facial-head movement videos from natural face-to-face conversations, enabling construction of a new test of non-acted expres- sion reading ability. The videos are much more complex than those in the MindReading DVD, including natural and potentially distracting movements such as head turns, hands over face, lips moving with speech, and other facial gestures that are hard to read by most people out of context. Using a subset with high NT-rater agreement, the best and worst recognition was when the Groden students looked at videos of themselves and their peers. Developing the Test The test has 90 questions, each showing two short clips: one shows a good example of a target expression and the other is similar to but not as representative of the target expres- sion (Fig.1). The target expression was one of: thinking, interested, unsure, happy, and smiling. Which is the best example of thinking Difficulty Figure 1: Test interface. See demo video on laptop. The test was administered in 2 sessions, 45 min- utes each. The questions and placement of clips/question were re-ordered randomly each time the test is taken. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% rater control subject interested unsure thinking smiling happy Chart 3: Correct answers per group by emotion. 50 60 70 80 90 100 50 60 70 80 90 100 Rater Control Subject Chart 2: Scatter of scores by group. Chart 1: Average scores for controls, raters and subject by type 0 2 4 6 8 10 12 14 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE Chart 4: Breakdown of ``Happy’’ questions 0 2 4 6 8 10 12 14 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 control - FALSE control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE Chart 5: Breakdown of ``Interested’’ questions 0 2 4 6 8 10 12 14 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 control - FALSE control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE Chart 6: Breakdown of ``Thinking’’ questions 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% control rater subject1 subject3 subject4 subject5 subject6 DVD staff stranger subject1 subject2 subject3 subject4 subject5 subject6 10 videos Spontaneous Posed Familiar Stranger Self Peers Staff Adult DVD Autistic Neuro-typical 50 videos 10 videos 10 videos 10 videos Generalization Levels Total of 90 questions
Transcript of Wearable Apparatus For Quantifying Social-Emotional ... · Wearable Apparatus For Quantifying...
Wearable Apparatus For QuantifyingSocial-Emotional Expression Recognition in Natural Face-to-Face InteractionsAlea Teeters, Rana El Kaliouby, Matthew Goodwin, Mia Shandell, Rosalind W. PicardMedia Laboratory, Massachusetts Institute of Technology & The Groden Center
ObjectivesThere is a need for tools that measure social-emotional expression recognition in face-to-face conversations to establish an accurate baseline of recognition ability and to evaluate the e�cacy of social interventions.
MethodsWe set up dyadic conversations ; each participant wore Self-Cam to record facial and head movements.
>>Data Collection Autistic-Typical pairs: 6 male students at the Groden Center with a diagnosis of mild to moderate autism spectrum disorder (ADOS) with anxiety disorder, ADHD and down syndrome interact with a sta� member twice a week for 10 minutes over 2 months. Typical-Typical pairs: 4 volunteers (2males, 2 females) from the MIT Media Lab were split into two pairs and asked to carry on a conversation for at least10 minutes.
>>Segmenting and Labeling the Videos From a total of 100 videos, 2 segmenters chose clips that show: disagreeing, thinking, unsure, confused, inter-ested, uninterested, angry, smiling, happy, sad, surprised, concerned, distracted.
Then, 10 independent labelers (who scored above 28/36 on Baron-Cohen’s Eyes Test) view each video and pick zero or more of the above labels. For 5 (out of 13) labels--thinking, interested, unsure, happy, and smiling--we chose the clips with the higest inter-rater reliability. These 180 clips form the test questions.
ResultsThe test was taken by 3 groups: 4 raters, 4 controls, and 5 out of the 6 Groden students (subject 2 was no longer available).
ConclusionsOur wearable self-cam technology successfully gathered facial-head movement videos from natural face-to-face conversations, enabling construction of a new test of non-acted expres-sion reading ability. The videos are much more complex than those in the MindReading DVD, including natural and potentially distracting movements such as head turns, hands over face, lips moving with speech, and other facial gestures that are hard to read by most people out of context. Using a subset with high NT-rater agreement, the best and worst recognition was when the Groden students looked at videos of themselves and their peers.
Developing the TestThe test has 90 questions, each showing two short clips: one shows a good example of a target expression and the other is similar to but not as representative of the target expres-sion (Fig.1). The target expression was one of: thinking, interested, unsure, happy, and smiling.
Which is the bestexample of
thinking
Di�culty
Figure 1: Test interface. See demo video on laptop.
The test was administered in 2 sessions, 45 min-utes each. The questions and placement of clips/question were re-ordered randomly each time the test is taken.
0%10%20%30%40%50%60%70%80%90%
100%
rater control subject
interested unsure thinking smiling happy
Chart 3: Correct answers per group by emotion.
50
60
70
80
90
100
50 60 70 80 90 100
Rater Control SubjectChart 2: Scatter of scores by group.
Chart 1: Average scores for controls, raters and subject by type
0
2
4
6
8
10
12
14
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE
Chart 4: Breakdown of ``Happy’’ questions
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
control - FALSE control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE
Chart 5: Breakdown of ``Interested’’ questions
0
2
4
6
8
10
12
14
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
control - FALSE control - TRUE rater - FALSE rater - TRUE Subject - FALSE Subject - TRUE
Chart 6: Breakdown of ``Thinking’’ questions
0%10%20%30%40%50%60%70%80%90%
100%
control rater subject1 subject3 subject4 subject5 subject6
DVD staff stranger subject1 subject2 subject3 subject4 subject5 subject6
10 videos
Spontaneous PosedFamiliar Stranger
Self Peers Sta� Adult DVDAutistic Neuro-typical
50 videos 10 videos 10 videos 10 videos
Generalization Levels
Total of 90questions
