AnInternationalICTR&DJournalSponsoredbyZTECorporation ... · ISSN1673-5188 CN34-1294/TN CODENZCTOAK...

ISSN 1673-5188CN 34-1294/ TNCODEN ZCTOAK

ZTECOMMUNICATIO

NS

VOLU

ME15

NUMBER

S2DEC

EMBER

2017

tech.zte.com.cn

ZTE COMMUNICATIONSDecember 2017, Vol. 15 No. S2

SPECIAL TOPIC:Motion and Emotion Sensing Drivenby Big Data

An International ICT R&D Journal Sponsored by ZTE Corporation

ZTE Communications Editorial Board

Members (in Alphabetical Order):

Chairman ZHAO Houlin: International Telecommunication Union (Switzerland)

Vice Chairmen ZHAO Xianming: ZTE Corporation (China) XU ChengZhong: Wayne State University (USA)

CAO Jiannong Hong Kong Polytechnic University (Hong Kong, China)

CHEN Chang Wen University at Buffalo, The State University of New York (USA)

CHEN Jie ZTE Corporation (China)

CHEN Shigang University of Florida (USA)

CHEN Yan Northwestern University (USA)

Connie ChangHasnain University of California, Berkeley (USA)CUI Shuguang University of California, Davis (USA)

DONG Yingfei University of Hawaii (USA)

GAOWen Peking University (China)

HWANG JenqNeng University of Washington (USA)LI Guifang University of Central Florida (USA)

LUO FaLong Element CXI (USA)MA Jianhua Hosei University (Japan)

PAN Yi Georgia State University (USA)

REN Fuji The University of Tokushima (Japan)

SONGWenzhan University of Georgia (USA)

SUN Huifang Mitsubishi Electric Research Laboratories (USA)

SUN Zhili University of Surrey (UK)

Victor C. M. Leung The University of British Columbia (Canada)

WANG Xiaodong Columbia University (USA)

WANG Zhengdao Iowa State University (USA)

WU Keli The Chinese University of Hong Kong (Hong Kong, China)

XU ChengZhong Wayne State University (USA)YANG Kun University of Essex (UK)

YUAN Jinhong University of New South Wales (Australia)

ZENGWenjun Microsoft Research Asia (USA)

ZHANG Chengqi University of Technology Sydney (Australia)

ZHANG Honggang Zhejiang University (China)

ZHANG Yueping Nanyang Technological University (Singapore)

ZHAO Houlin International Telecommunication Union (Switzerland)

ZHAO Xianming ZTE Corporation (China)

ZHOUWanlei Deakin University (Australia)

ZHUANGWeihua University of Waterloo (Canada)

CONTENTSCONTENTS

Submission of a manuscript implies thatthe submitted work has not been publishedbefore (except as part of a thesis or lecturenote or report or in the form of anabstract); that it is not under considerationfor publication elsewhere; that itspublication has been approved by allco-authors as well as by the authoritiesat the institute where the work has beencarried out; that, if and when themanuscript is accepted for publication, theauthors hand over the transferablecopyrights of the accepted manuscript toZTE Communications; and that themanuscript or parts thereof will not bepublished elsewhere in any languagewithout the consent of the copyrightholder. Copyrights include, without spatialor timely limitation, the mechanical,electronic and visual reproduction anddistribution; electronic storage andretrieval; and all other forms of electronicpublication or any other types ofpublication including all subsidiary rights.

Responsibility for content rests onauthors of signed articles and not on theeditorial board of ZTE Communications orits sponsors.

All rights reserved.

Guest EditorialREN Fuji and GU Yu

01

How Do Humans Perceive Emotion?LI Wen03

Multimodal Emotion Recognition with Transfer Learningof Deep Neural Network

HUANG Jian, LI Ya, TAO Jianhua, and YI Jiangyan

23

Emotion Judgment System by EEG Based on Concept Baseof EEG Features

Mayo Morimoto, Misako Imono, Seiji Tsuchiya, and Hirokazu Watabe

11

Emotion and Cognitive Reappraisal Based on GSRWearable Sensor

LI Minjia, XIE Lun, WANG Zhiliang, and REN Fuji

18

Emotion Analysis on Social Big DataREN Fuji and Kazuyuki Matsumoto

30

ISSN 1673-5188CN 34-1294/ TNCODEN ZCTOAK

tech.zte.com.cn

ZTE COMMUNICATIONSDecember 2017, Vol. 15 No. S2

SPECIAL TOPIC:Motion and Emotion Sensing Drivenby Big Data

An International ICT R&D Journal Sponsored by ZTE Corporation Special Topic: Motion and Emotion Sensing Driven by Big Data

ZTE COMMUNICATIONSVol. 15 No. S2 (Issue 60)QuarterlyFirst English Issue Published in 2003Supervised by:Anhui Science and Technology DepartmentSponsored by:Anhui Science and Technology InformationResearch Institute and ZTE CorporationStaff Members:Editor-in-Chief: CHEN JieExecutive AssociateEditor-in-Chief: HUANG XinmingEditor-in-Charge: ZHU LiEditors: XU Ye, LU Dan, ZHAO LuProducer: YU GangCirculation Executive: WANG PingpingAssistant: WANG KunEditorial Correspondence:Add: 12F Kaixuan Building,329 Jinzhai Road,Hefei 230061, ChinaTel: +86-551-65533356Fax: +86-551-65850139Email: [email protected] and Circulated(Home and Abroad) by:Editorial Office ofZTE CommunicationsPrinted by:Hefei Tiancai Color Printing CompanyPublication Date:December 25, 2017Publication Licenses:

Annual Subscription:RMB 80

ISSN 1673-5188CN 34-1294/ TN

CONTENTSCONTENTS

Review

Security Enhanced Internet of Vehicles withCloud⁃Fog⁃Dew Computing

MENG Ziqian, GUAN Zhi, WU Zhengang, LI Anran, and CHEN Zhong

47

Measuring QoE of Web Service with Mining DNS Resolution DataLIU Yongsheng, GU Yu, WEN Xiangjiang, WANG Xiaoyan, and FU Yufei

38

An Improved K⁃Means Algorithm Based on Initial ClusteringCenter Optimization

LI Taihao, NAREN Tuya, ZHOU Jianshe, REN Fuji, and LIU Shupeng

43

Research Paper

Random Forest Based Very Fast Decision Tree Algorithmfor Data Stream

DONG Zhenjiang, LUO Shengmei, WEN Tao, ZHANG Fayang, and LI Lingjuan

52

Deep Neural Network⁃Based Chinese Semantic Role LabelingZHENG Xiaoqing, CHEN Jun, and SHANG Guoqiang

58

Roundup

Table of Contents for Volume 15, 2017 I

Motion and Emotion Sensing Driven by Big DataMotion and Emotion Sensing Driven by Big Data

▶ REN FujiProf. REN Fuji received the Ph.D. de⁃gree in 1991 from Faculty of Engi⁃neering, Hokkaido University, Japan.He worked at Computer Service Cor⁃poration (CSK), Japan, where he wasa chief researcher of natural languageprocessing (NLP) from 1991. Since2001, he has been a professor of theFaculty of Engineering, TokushimaUniversity, Japan. He was the direc⁃

tor of Information Solution, Tokushima University. He isthe president of International Advanced Information Insti⁃tute, Japan. He is also an advisor professor or visiting pro⁃fessor in New Mexico State University, Florida Internation⁃al University, Harvard University, Hefei University ofTechnology, Beijing University of Posts and Telecommuni⁃cations, Nanjing University, Tongji University, and Xi’anJiaotong University. His current research interests includenatural language processing, artificial intelligence, affec⁃tive computing, and emotional robot. He is Fellow ofJFES, IEICE, CAAI, and HTC, and a member of The Ja⁃pan Federation of Engineering Societies. He is a Chang Ji⁃ang Scholar Endowed Chair Professor of the Chinese Min⁃istry of Education, a winner of the Chinese Academy of Sci⁃ence Fund for Distinguished Overseas Young Scholars,and Chinese“1000 Talents Plan”Distinguished Expert.


Special Topic

M

ZTE COMMUNICATIONSZTE COMMUNICATIONS 01December 2017 Vol.15 No. S2

otion is a significant symbol of living beings, and emotion is aunique feature of human beings. With the rapid expansion of humansociety, there is a growing need for real ⁃ time motion and emotionsensing in various real⁃world applications such as intrusion detec⁃

tion, patient sleep monitoring, elder emotion companion and autism treatment. Todate, vision⁃based and sensor⁃based sensing methods are the dominating solutionsin this area. The former usually employs surveillance cameras to take footage, fromwhich gradient ⁃ based features are extracted for motion and emotion recognition,while the latter explores virtual footprints caused by human beings from specificsensors for recognizing human activities and moods.

Though significant progress has been obtained, the related research still facesseveral challenges, e.g., the availability issue (specialized hardware), the reliabilityissue (light and line⁃of⁃sight (LOS) restrictions), and the privacy issue (physical con⁃tacts and feeling violated when being watched). Therefore, researchers are strug⁃gling for new paradigms to revolutionize the traditional solutions. The rapid develop⁃ment of wireless communications and embedded platforms has greatly acceleratedthe approaches for the Internet of Things (IoT) and envisioned a new era of ubiqui⁃tous computing. The ever⁃increasing smart devices surrounding our daily lives haveoffered a great opportunity for building a big data driven system that can recognizehuman⁃centric motions anytime anywhere.

This special issue deals with the overview, technology, and applications of usingbig data from various IoT devices for motion and emotion sensing, and aims to stim⁃ulate research and development of related areas by providing a unique forum forscientists, engineers, broadcasters, manufacturers, software developers, and otherrelated professionals . The topics addressed in this special issue are diversified, in⁃cluding an overview of current research progresses on IoT, real⁃world applications,security issues, etc. The call⁃for⁃papers for this special issue attracted a number ofexcellent submissions. After two⁃round reviews, seven papers were selected for pub⁃lication. These papers are organized in three groups. The first group comprises oneoverview paper that addresses the most fundamental question in the affective com⁃puting, i.e., how humans perceive emotion. The second group consists of four pa⁃pers addressing the emotion sensing issues by leveraging big data from different per⁃spectives, e.g., electroencephalogram (EEG) sensors, galvanic skin reaction (GSR)wearable sensors, social big data, etc. The last group comprises two papers thatpresent basic methodologies to ensure reliable and accurate big data for emotionsensing.

The first paper“How Humans Perceive Emotion?”directly studies the most fun⁃damental issue of emotion perceiving and reviews behavioral as well as neural find⁃ings in human emotion perception, including facial emotion perception, olfactoryemotion perception, multimodal emotion perception, and the time course of emotionperception. The knowledge of how humans perceive emotion will help bring artifi⁃cial intelligence strides closer to human intelligence.

The most⁃commonly used data sources in affective computing are psychological

▶ GU YuProf. GU Yu received his D.E. degreefrom the Department of Computer Sci⁃ence, University of Science and Tech⁃nology of China, China in 2010 (withthe Excellent Award of Chinese Acad⁃emy of Sciences). He received his B.Eng degree in 2004, from the SpecialClasses for the Gifted Young, Univer⁃sity of Science and Technology of Chi⁃na. From Feb. 2006 to Aug. 2006, he

was an intern in Wireless Network Group, Microsoft Re⁃search Asia, China. From Dec. 2007 to Dec, 2008, he wasa visiting scholar with the Department of Computer Sci⁃ence, Graduate School of Systems and Information Engi⁃neering, University of Tsukuba, Japan. From Nov. 2010 toOct. 2012, he worked in the National Institute of Informat⁃ics (Japan) as a JSPS research fellow. He is a full⁃time pro⁃fessor with School of Computer and Information, Hefei Uni⁃versity of Technology (HFUT), China and was awarded“Huangshan Mountain Yong Scholar”in 2012. His re⁃search interests include effective computing and pervasivecomputing, especially context ⁃ aware mobile sensing. Hereceived the Excellent Paper Award at IEEE Scalcom2009 and Best Paper Award at NLP⁃KE 2017. He is a se⁃nior member of IEEE and a member of ACM.

sensors like EEG and electrocardiogram (ECG) sensors. Thesecond paper“Emotion Judgment System by EEG Based onConcept Base of EEG Features”proposes an emotion judg⁃ment system by using the EEG feature concept base with prem⁃ise of noises included. The proposed system is testified on aprivate EEG dataset and compared with the traditional supportvector machine (SVM) classifier. The results proves the effi⁃ciency of the proposed system.

The third paper“Emotion and Cognitive Reappraisal Basedon GSR Wearable Sensor”studies a similar problem with a dif⁃ferent data source, i.e., GSR wearable sensors. Based on theauthors’previous work, they establish a novel GSR emotionsensing system for predicting the emotional state transition andconsidering the correlation between GSR signals and emotions.

The fourth paper“Multimodal Emotion Recognition withTransfer Learning of Deep Neural Network”tackles the emo⁃tion recognition problem from the aspect of multimodality in⁃cluding both images and audios, where audio is encoded usingdeep speech recognition networks with 500 hours’speech andvideo is encoded using convolutional neural networks (CNNs)with over 110,000 images. The following experiments revealseveral interesting observations. For example, audio featuresextracted from deep speech recognition networks can achievebetter performance than handcrafted audio features.

Another emerging source for emotion analysis is data fromthe fastly expanding social networks. The fifth paper“EmotionAnalysis on Social Big Data”describes a new method of emo⁃tion analysis based on social big data, e.g., multilingual webcorpora and its annotated emotion tags. To handle the collect⁃

ed social big data, the authors design an emotion analysis mod⁃el that integrates the high⁃quality emotion corpus and the auto⁃matic ⁃ constructed corpus created in their past studies, andthen tries to analyze a large⁃scale corpus consisting of Twittertweets based on the model. The proposed system is evaluatedvia time ⁃ series analysis on the large ⁃ scale corpus and modelevaluation, and demonstrated effective over previous solutions.

Furthermore, the reliability of social big data is also anemerging problem. Therefore, the sixth paper“Measuring QoEof Web Service with Mining DNS Resolution Data”proposes anovel solution called the First Webpage Time (FWT) algorithmto measure the QoE of the web service. The proposed FWT al⁃gorithm is analyzed in theory that its precision is guaranteed.The experiments based on the ISP’s DNS resolution data arecarried out to evaluate the proposed FWT algorithm.

Other than reliable data sources, effective classifiers are al⁃so important to the emotion analysis system. Therefore, the lastpaper“An Improved K⁃means Algorithm Based on Initial Clus⁃tering Center Optimization”targets on the widely ⁃ used K ⁃means algorithm and proposes an effective algorithm to selectthe initial clustering center for eliminating the uncertainty ofthe central point selection.

As we conclude the introduction of this special issue, wewould like to thank all the authors for their valuable contribu⁃tions, and we express our sincere gratitude to all the reviewersfor their timely and insightful expert reviews. It is hoped thatthe contents in this special issue are informative and usefulfrom the aspects of technology, standardization, and implemen⁃tation.

Special Topic


ZTE COMMUNICATIONSZTE COMMUNICATIONS02 December 2017 Vol.15 No. S2

How Do Humans Perceive Emotion?How Do Humans Perceive Emotion?LI Wen(Department of Psychology and Program of Neuroscience, Florida State University, Tallahassee, FL 32303, USA)

Abstract

Emotion carries crucial qualities of the human condition, representing one of the major challenges in artificial intelligence. Re⁃search in psychology and neuroscience in the past two to three decades has generated rich insights into the processes underlyinghuman emotion. Cognition and emotion represent the two main pillars of the human psyche and human intelligence. While the hu⁃man cognitive system and cognitive brain has inspired and informed computer science and artificial intelligence, the future is ripefor the human emotion system to be integrated into artificial intelligence and robotic systems. Here, we review behavioral and neu⁃ral findings in human emotion perception, including facial emotion perception, olfactory emotion perception, multimodal emotionperception, and the time course of emotion perception. It is our hope that knowledge of how humans perceive emotion will helpbring artificial intelligence strides closer to human intelligence.

emotion perception; faces; smells; time course; neural basisKeywords

DOI: 10.3969/j. issn. 16735188. 2017. S2. 001http://kns.cnki.net/kcms/detail/34.1294.TN.20171226.0940.002.html, published online December 26, 2017

W

This work is partially supported by Japan Society for the Promotion ofScience (JSPS, 16K00311).

1 Introductionhile computers and artificial intelligence haveassumed the supreme power of computationand“reasoning” (in some aspects such aschess and go) that excels even the finest hu⁃

man mind, they crumble at the simplest task that a toddler caninstinctually perform—emotion perception. Whether they arepurposeful products of evolution to serve adaptive functions ormere vestiges as energy spillover during physiological shifts[1], [2], emotional expressions are salient social cues in every⁃day interactions, communicating one’s emotional states and ac⁃tion tendencies to conspecifics [3]- [5]. Social bonding andcommunal behavior via emotion (known as“emotional conta⁃gion”; Hatfield, 1993) have the capacity to reach millions ofpeople and last over decades [6], [7]. Simply put, emotion is acritical ingredient of the human condition, imbuing it with rich⁃ness and sensibility [8]. Such importance and utility of emotionhas compelled artificial intelligence to incorporate emotionand feelings into neural networks and robotic systems [9].

To the extent that such efforts are encouraging and somepromises are on the horizon, emotion communication is such acomplex system that all endeavors so far have fallen short.While artificial face recognition has advanced greatly in thefield, reaching remarkable accuracy and speed, artificial emo⁃

tion perception still lags behind. Emotional expressions are of⁃ten transmitted and synchronized with such potency and speedthat any other stimuli would pale in comparison [10], [11]. Forinstance, encoding of facial expressions or emotional gesturesin the receiver can consummate as early as 100 ms [12]-[17],preceding the latency for structural encoding of faces (~ 170ms) [18], [19].

This paper will discuss how this seemingly extraordinaryfeat is achieved, quite effortlessly, in humans, in hopes thatsuch knowledge will inform and inspire the research of artifi⁃cial emotion perception. Given its obvious advantage and domi⁃nance in emotion communication, the discussion will focus onfacial emotion perception. Nevertheless, emotion is communi⁃cated through multiple sensory channels. Chemical informa⁃tion, such as smells and pheromones, may provide pivotal andsometimes indispensable information to mediate emotion com⁃munication. Therefore, the review will discuss emotion percep⁃tion via the olfactory channel and how multiple modal cues arecombined in emotion perception. To account for the remark⁃able speed of emotion perception, the time course of emotionperception will be described with a special focus on early pro⁃cesses. The review will end with a discussion of sensory corti⁃cal encoding of emotion, which could provide particularly use⁃ful insights into computation modeling of emotion perception.

2 Visual Emotion PerceptionFaces are inherently salient, emotion ⁃ relevant stimuli such

Special Topic


that even“neutral”faces would be emotionally charged due toits race, gender, eye gaze, attractiveness, and so on. According⁃ly, faces often attain preferential perception compared to otherobjects [20]. Highlighting an innate advantage in face percep⁃tion, human newborns prefer tracking faces to non⁃face objects[21], [22]. Infant monkeys reared with no exposure to any facesalso prefer looking at human and monkey faces than non⁃faceobjects [23]. Notably, these stimuli are controlled for basic vi⁃sual properties, such as contrast, complexity and spatial fre⁃quency, to exclude any possible confounds due to physical dis⁃parities.

Dubbed as the“face in the crowd”effect [24], faces contain⁃ing threat emotions such as anger and fear receive particularlyprivileged perceptual analysis, often“popping out”from acrowd of faces containing neutral or happy emotions, showingfaster reaction times and higher accuracy in face detection anddiscrimination [25], [26]. This effect is particularly consistentacross studies using schematic faces but relatively controver⁃sial when photos of real faces are involved [27], which couldbe ascribed to the fact that schematic faces are generally de⁃prived of information related to race, gender, attractivenessand so on, i.e. the factors that are rich in real face photos butimpoverished in schematic faces [27]. Indeed, studies that at⁃tempt to reduce such effects tend to support this face in thecrowd effect [28] as opposed to studies lacking such face con⁃trol [29], [30].

Recent studies further suggest that this acute emotion per⁃ception not only differentiates threat from non ⁃ threat (neutralor positive) stimuli (along general dimensions of affective va⁃lence/arousal; Russell, 1980), but is also capable of dissociat⁃ing individual basic emotions (Ekman, 1992), even within thedomain of threat (e.g., fear, disgust and anger). During basicperception, fear purportedly elicits an immediate“stop ⁃ look ⁃and ⁃ listen”response to facilitate sensory acquisition in orderto guide action (e.g., fight or flight; Gray, 1987), while disgustprovokes instant sensory rejection to prevent poison or contami⁃nation from entering the organism [31]. These opposing senso⁃ry tendencies align with the contrary biomechanical propertiesin facial expressions of fear and disgust. That is, wheareas fear⁃ful faces are characterized by widened eyes and nostrils, whichaugment visual and oflactory sensory intake, disgust faces arerepresented by narrowed eyes and nostrils, which restrict sen⁃sory intake (Susskind et al., 2008). Critically, these opposingsensory responses have been repeatedly demonstrated in ourlab [32]-[34] and others (Liu et al., 2015); fear⁃evoking scenes/faces enhance whereas disgust ⁃ evoking scenes/faces suppressvisual event⁃related potentials (ERPs; e.g., the P1 component,an early visual ERP peaking around 100 ms) and concomitantextrastriate cortical activity.These findings highlight the eco⁃logical adaptiveness inherent in emotion perception, promotingbiologically appropriate actions with minimal delay.

Neuroscience research in the past few decades has providedimportant insights into mechanisms and processes involved in

encoding and recognizing emotional expressions. As summa⁃rized in recent meta⁃analyses of neuroimaging studies, substan⁃tial evidence implicates limbic/paralimbic structures, especial⁃ly the amygdala and, to a lesser extent, the anterior cingulatecortex and ventral medial prefrontal cortex/orbitofrontal cortex,in processing facial expressions [35], [36]. These regions ap⁃pear to be nodes shared by the emotional brain [37] and the so⁃cial brain (Adolphs, 2009), akin to their roles in processing so⁃cially relevant emotional information. Notably, these regionsare also responsive to emotional vocalization (e.g., laughter andscreams [38]; and emotional touch [39], [40], representing acore system supporting amodal, abstract emotion analysis andevaluation. As for the processing of specific (vs. general) emo⁃tions, data to date are not as conclusive. Nevertheless, fairlyclear consensus has emerged for fear and disgust facial expres⁃sions, which reliably activate the amygdala and insula, respec⁃tively [35], [36]. In keeping with that, perception of fear anddisgust vocalizations also depends on these same structures[41], [42].

While emotion research has focused primarily on limbic/paralimbic structures, accruing evidence also isolates a highlyassociative, heteromodal sensory cortex—the superior tempo⁃ral sulcus (STS) [43]; STS is a conventional multisensory zone,involved in vision, audition and somatosensation [44]. Impor⁃tantly, it has been implicated in the social neural network [45],processing sophisticated social cues to infer other people’smental states (e.g., theory of mind) [46], [47]. Accordingly, theSTS is found to play a critical role in the perception of facialexpressions, especially dynamic ones [48]- [50]. Furthermore,the STS exhibits specialized response patterns for facial expres⁃sions of the basic emotions (anger, disgust, fear, sadness, hap⁃piness, and surprise; Ekman, 1992) and supports highly inte⁃grated analysis of subtle differences in emotional expressions[47]. Representing a key voice⁃processing area [51], the STS al⁃so participates in assessing emotional vocalizations [52]. Last⁃ly, the importance of the sensory cortex in supporting emotionperception, independently of limbic input, has been increasing⁃ly recognized [53]-[55].

3 Olfactory Emotion PerceptionAs summarized above, emotional expressions are typically

considered to be transmitted through faces and (to a lesser ex⁃tent) posture, vocalizations, and touch, with research interestpredominantly dedicated to facial expressions. However, arechemical senses involved in emotion communication as well?Are chemical signals (chemosignals) of emotion processed sim⁃ilarly as physical signals?

A large body of literature documents emotion communica⁃tion via chemical stimuli (odors and pheromones) in non ⁃pri⁃mate animals [56], [57]. In these animals, olfaction is the mostcrucial sensory channel, purportedly principally involved in ef⁃fectively detecting, locating, and identifying reward and threat

Special Topic

How Do Humans Perceive Emotion?LI Wen


in the environment [58]. Accordingly, vital biological informa⁃tion is transmitted among conspecifics via olfaction, informingfood or poison, mate or predator. In humans, olfaction isdeemed as a minor sensory system, and humans are consideredmicrosmatic [59], presumably due to Lamarckian disuse of theolfactory sense. Nevertheless, this microsmia notion has beenchallenged by recent work, documenting the remarkable capac⁃ity of human olfaction at both neuronal and behavioral levels[60]-[63]. Furthermore, the olfactory system is intimately asso⁃ciated with the emotion system: odors provoke potent emotion⁃al responses in people [64], [65], and the neuroanatomy of ol⁃faction and emotion is intricately connected as in macrosmaticanimals [66], [67]. Indeed, infinitesimal amounts of odors (aslow as 7 ppt) can be processed by the human olfactory system(albeit subliminally), thereby modulating affective processingof faces [68], [69]. Lastly, the human body constantly secreteschemicals, which vary in intensity and chemical compositionwith internal/endocrine states and interactions with residentbacteria (primarily, in axillary areas and genitalia [70]. Owingto this close association with the host’s emotional and physio⁃logical states, these chemical excrements can carry potent in⁃formation about momentary emotion and intended/prepared ac⁃tion. Overall, the special faculty of olfaction promises the sig⁃nificance of this chemical sensory system in social communica⁃tion of emotion.

Accruing evidence indeed suggests that humans performsimilar chemical communication of emotion as other animals.Human body odors and fluids carry certain genetic informationsuch that by smelling these chemicals, the receiver can deter⁃mine his/her genetic compatibility [71], [72] and kinship [73]-[75] with the sender, thereby preventing inbreeding while en⁃hancing nepotism. Similar to chemosensory⁃based avoidance ofsick conspecifics in other mammals [76], a new study showsthat when people are sick, their body odors change, becomingmore unpleasant and unhealthy to other people [77]. In addi⁃tion, people can detect or differentiate elevated arousal [78],[79] as well as specific emotions (happiness, fear, disgust andanxiety; Ackerl, Atzmueller, & Grammer, 2002; Chen & Havi⁃land ⁃ Jones, 2000; de Groot, Smeets, Kaldewaij, Duijndam, &Semin, 2012; Pause, Ohrt, Prehn, & Ferstl, 2004; Prehn⁃Kris⁃tensen et al., 2009) by smelling axillary sweat. Interestingly, fa⁃miliarity between the sender and the receiver enhances recog⁃nition of emotion conveyed in body odors (Ackerl, Atzmueller,& Grammer, 2002; Chen & Haviland ⁃ Jones, 2000; de Groot,Smeets, Kaldewaij, Duijndam, & Semin, 2012; Pause, Ohrt,Prehn, & Ferstl, 2004; Prehn⁃Kristensen et al., 2009).

In terms of underlying neural basis of chemosignaling ofemotion, evidence is fairly scarce. Nevertheless, the extantneuroimaging data largely converge on limbic participation inemotion communication via chemosignals [80], conforming toemotion communication via other sensory signals. Specifically,body odors conveying potential threat evoke strong response inthe amygdala [78], [81]- [83]. In general, the extant literature

combined with long⁃standing animal research suggests that hu⁃man olfactory emotion communication represents a highly valu⁃able research subject and, potentially, an emerging frontier ofthe field.

4 Multisensory Integration of EmotionalSignalsAs emotional expressions are communicated via multiple

senses, and very often simultaneously (e.g., a terrified face be⁃ing accompanied by a shaky voice, tense posture, and quitelikely, a particular body odor), a natural question becomes howmultisensory emotional signals are integrated in social commu⁃nication. For either neural activity or consequent behavior, anorganizing principle is that inputs from multiple senses con⁃verge and interact in a variety of brain structures, supportinghighly coordinated responses [84], [85]. In fact, organisms asprimitive as a progenitor cell integrate information from multi⁃ple senses to optimize perception; moreover, this synergy is es⁃pecially prominent with minimal sensory input, facilitating sig⁃nal processing in impoverished or ambiguous situations(known as the principle of“inverse effectiveness”[86]. Con⁃ceivably, multisensory integration of emotional expressionswould afford a special ecological advantage by facilitating com⁃munication of salient information, especially when such activi⁃ty is impeded by various sensory barriers (e.g., darkness, dis⁃tance or background noise) or suppressed in special situations(e.g., communicating with a hostage under close watch). Whilepertaining primarily to integration between visual and auditorysenses, research in the past decade has shed some first light onthe mechanisms underlying multisensory integration of emo⁃tional expressions [87], [88]. Akin to its role in processing ofemotion and its multimodal connections (via dense bidirection⁃al fibers linking all sensory cortices) [89], the amygdala hasemerged as a key convergence/integration area in this litera⁃ture. Furthermore, the STS (particularly, the posterior STS) hasalso been isolated as a key site for multisensory convergence ofemotional expressions [87], [90]. As mentioned above, the pos⁃terior STS is long known as a key multisensory convergencezone, largely due to the multimodal (visual, auditory and so⁃matosensory) neurons in this region and dense fibers connect⁃ing it to different sensory cortices [84], [91]- [93]. Also, func⁃tional connectivity analysis suggests that the STS not only en⁃gages in integrating emotional expressions across modalities(between faces and voices) but also gates synthesized sensoryinput to the amygdala [94]. Given the modest size of this litera⁃ture, however, it remains unclear whether multisensory integra⁃tion of discrete emotional expressions would recruit distinct orshared convergence areas and mechanisms.

Another problem yet to be explored is integration of emotion⁃al expressions between physical and chemical senses. Awealth of behavioral data evinces active visual and olfactory in⁃teraction in information processing [95]-[98]. Evidence further

Special Topic



suggests that synthetic or body odors can improve perceptionof visual social signals including facial expressions [14], [99]-[101]. However, neural evidence in this regard is rather limit⁃ed, and preliminary evidence suggests that the insula and orbi⁃tofrontal cortex are involved in integrating emotional signals infaces and synthetic or body odors [102]- [104]. However, dueto the relative anatomical segregation between the physicaland chemical sensory systems, it is likely that a fairly intricateneural network would be recruited in integrating social signalsof emotion between these two systems. Indeed, in contrast tocrossmodal auditory⁃visual integration that may takes place atearly sensory corticies [105], visuo⁃olfactory integration of emo⁃tion favors a top⁃down (vs. bottom⁃up) account of crossmodalintegration in higher⁃order brain areas [104].

5 Stages of Emotion PerceptionAs aforementioned, emotion perception is marked by its ex⁃

traordinary speed, taking as little as 100 ms to isolate facial ex⁃pressions. How does the process of emotion perception unfoldover time? Starting from a highly prominent two⁃stage model ofearly“quick⁃and⁃dirty”and delayed, elaborate processing ofthreat information [106], the emotion literature has expandedto support a complex system involving multiple stages and pro⁃cesses, mediated by distributed, parallel neural pathways [11],[107], [108]. Several influential cognitive theories converge ona parsimonious model of three stages—an orienting mode, aprimal mode and a metacognitive model—operating in se⁃quence over time [109], [110]. In general, predominant chargesfor these three stages are feature detection, significance evalua⁃tion, and conscious threat perception, respectively [111],[112]. During the first stage, an external stimulus registers withthe“feature detectors”in a nonconscious, automatic fashion.These detectors isolate signal features of biologically signifi⁃cant stimuli, which then triggers the nonconscious“signifi⁃cance evaluator.”Confirmation from the significance evaluatorturns on the third stage: controlled, strategic processing of thestimulus, generating conscious threat perception. Notably,Ohman’s model also emphasizes that autonomic arousal is di⁃rectly activated by feature detectors, which provides input to fa⁃cilitate significance evaluation and conscious threat percep⁃tion.

Empirical evidence from human neuroscience researchaligns with this multi ⁃ stage view. Brain electrophysiological(mainly ERP) research has leveraged on its precise temporalresolution to delineate the time course of information process⁃ing of emotional stimuli on the scale of milliseconds. Findingsfrom this research implicate three temporal stages of emotionprocessing [11], [107], [113], [114]. The first stage, indexed bythe P1 component, represents sensory processing of emotionalstimuli in the low ⁃ level, occipital visual cortex. The secondstage, indexed by the N1/N170 components (onset ~170 ms),entails intermediate ⁃ level, configural perceptual analysis in

the temporal visual cortex. The third stage, indexed by the P3/P300 and late positive potential (LPP) components (~300 msand beyond), reflects high ⁃ level, cognitive and motivationalprocesses. During this stage, emotion processing engages mem⁃ory⁃based, goal⁃oriented operations, often culminating in con⁃scious perception of the stimuli and volitional behavioral re⁃sponse. Broadly speaking, this sequence of electrophysiologi⁃cal events corresponds really closely to the three main stagesof the cognitive models above.

Pertinent to the perception of threat specifically, a recentstudy in our lab acquired fear detection rates and ERPs toparametrically varied levels of fearful expressions along a mor⁃phing continuum [115]. To provide further insights into the spe⁃cific cognitive mechanisms involved at different stages inthreat perception, we decomposed the threat processing bycombining psychometric and neurometric modeling. Buildingon the psychometric curve marking fear perception thresholds(e.g., detection, sub⁃ and supra⁃ threshold perception), neuro⁃metric model fitting identified four key operations along the in⁃formation processing stream (Fig. 1). Unfolding in sequencefollowing face presentation, these four psychological processesare: 1) swift, coarse categorization of fear versus neutral stimuli(~100 ms, indexed by the P1), 2) detection of fear by pickingup minute but psychologically meaningful signals of fear (~320ms, indexed by the P3), 3) valuation of fear signal by trackingsmall distances in fear intensity, including subthreshold fear(400-500 ms, indexed by an early subcomponent of the LPP),and lastly 4) conscious awareness of fear supporting visibilityof suprathreshold fear (500-600 ms, indexed by a late subcom⁃ponent of the LPP). Furthermore, as the processes became pro⁃gressively refined over time, they were also increasingly linkedto behavioral performance (i.e., fear detection rates; Fig. 1d,bottom row). Specifically, from the first to the last operations,within⁃subject brain⁃behavior association grew from no associa⁃tion, to weak, then moderate, and finally strong, respectively.

Overall, these findings provide specific descriptions andtemporal profiling of threat processing stages. The first opera⁃tion—broad threat ⁃ non ⁃ threat categorization—would corre⁃spond to the orienting mode in threat processing, which auto⁃matically tags the stimuli as threat or non ⁃ threat. Such grosscategorization (at the P1 window) concurs with standard objectcategorization (e.g., natural vs. domestic scenes) [116]. Thisfinding also aligns with the notion that emotional stimuli canelicit rapid emotion categorization based on automatic, bottom⁃up sensory input [117], [118], coinciding with Ohman’s idea of“feature detectors”that isolate threat ⁃ relevant signal features[111], [112]. This significance detection then activates salience⁃driven bottom⁃up attention and the brain’s salience network,which switches on other networks to start resource allocation(via attention and working memory) and goal⁃driven processesin the subsequent stages [119]-[121].

The second and third operations—threat detection and valu⁃ation—would largely fall into the primal mode as the interme⁃

Special Topic



diate⁃ level threat analysis. As illustrated in Fig. 1, the neuraldetection threshold aligns with the inflection point (25% fear)of the psychometric function, and the strength of this neural re⁃sponse is significantly (though only weakly) predictive of feardetection rates, suggesting somewhat reliable threat detectionat this stage. The third operation is more sophisticated and ad⁃vanced, linearly tracking the intensity of fearful expressionsand directly predicting behavior performance (r = 0.41). Thelast operation brings about conscious awareness, correspond⁃ing closely to the metacognitive mode, where consciousness ofthreat emerges and conscious processes ensue. In keeping withthat, this last operation accounts for a remarkable 31% of the

total variance of the behavioral output.Compared to the later operations (especially threat valuation

and awareness), the first operation (threat tagging) does notshow a relation with the behavior. This finding underscores theidea that the orienting mode is likely to be elusive to behavior⁃al observation. Many creative paradigms (e.g., emotionalStroop, dot⁃probe, visual cueing, and visual search) have beenused to isolate early operations in threat processing, but aspointed out early in the field, behavioral measures from thesetasks are inevitably confounded by operations from multiplestages [122]. By virtue of the rapid development of neuroscien⁃tific methods, especially brain electrophysiology technologies,

▲Figure 1. The time course of human emotion perception. a) The GFP demonstrates five critical ERPs evoked by faces. b) Evoked P1 at the occipitalmidline by the neutral face and six levels of fear (15%-45% in increments of 6%). c) Three ERPs evoked at the parietal midline by the faces. d) Psycho⁃metric and neurometric modeling of fear detection performance and ERPs in a fear detection task maps out four key operations unfolding in sequence,emotion categorization, detection, valuation and conscious awareness. Adapted from Forscher et al., 2016.

GFP: global field power LPP: late positive potential

4.6

Time (ms)a)

2.5

GFP(μ

V)

3.03.54.0

1.52.0

0.51.0

P300P1 eLPP

ILPP

N170

-200 0 200 400 600Oz

(μV)

5432

1

2%15%21%27%33%39%45%

Time (ms)b)

0 200100 300-100

P1

Pz(μV

) 4

2

6

Time (ms)c)

0 200 600-100 400

LPPLate

Early

P300

%Fear0 20 40

Categorization

P1(μV

)

4.8

4.4

0.5

-0.5

0-0.25

0.25μV

Detection

%Fear0 20 40

3.8P300(μ

V)

4.0

μV73.50-3.5-7

%Fear0 20 40

5.0

EarlyL

PP(μV

)

4.5

5.5

μV0.751.5

0

-1.5-0.75

Valuation Awareness

%Fear0 20 40

LateL

PP(μV

)

6.56.0

5.5

5.0

μV0.250.5

0-0.25-0.5

1.0

Fearde

tection

rates 0.8

0.60.40.20.0

P1 amplitude (μV)121086420

r = 0.09, ns

1.0

Fearde

tection

rates 0.8

0.60.40.20.0

P300 amplitude (μV)121086420

r = 0.14*

1.0

Fearde

tection

rates 0.8

0.60.40.20.0

Early LPP amplitude (μV)121086420

r = 0.41***

1.0

Fearde

tection

rates 0.8

0.60.40.20.0

Late LPP amplitude (μV)121086420

r = 0.56***

Time ~100 ms ~320 ms 400~500 ms 500~600 msd)

Special Topic



relatively pure measures of the orienting mode have become vi⁃able.

6 ConclusionsThe field of artifical intelligence has progressed in leaps and

bounds over the past decade. How artifical intelligence can be⁃come truly intelligent, taking possession of the human condi⁃tion, has been the holy grail of the field. In the search of the hu⁃man psyche, knowledge of reasoning and cognition had preced⁃ed the understanding of emotion. Now, growing insights intohuman emotion and emotional processes, such as emotion per⁃ception, have issued cordial invitaitons to computer scientiststo adopt emotion⁃related models and paradigms into artifical in⁃telligence. The future is ripe, tarry not.

References[1] C. R. Darwin, The Expression of the Emotions in Man and Animals, 1 ed. London,

UK: John Murray, 1872.[2] A. F. Shariff and J. L. Tracy,“What are emotion expressions for?”Current Direc⁃

tions in Psychological Science, vol. 20, no. 6, no. 395- 399, Dec. 2011. doi:10.1177/0963721411424739.

[3] C. E. Izard, The Face of Emotions. New York, USA: Appleton⁃Century ⁃Crofts,1971.

[4] A. J. Fridlund, Human Facial Expression: An Evolutionary View. San Diego,USA: Academic Press, 1994.

[5] R. E. Kraut and R. E. Johnston,“Social and emotional messages of smiling: anethological approach,”Journal of Personality and Social Psychology, vol. 37, no.9, pp. 1539-1553, 1979.

[6] A. D. Kramer, J. E. Guillory, and J. T. Hancock,“Experimental evidence of mas⁃sive⁃scale emotional contagion through social networks,”Proceedings of the Na⁃tional Academy of Sciences of the United States of America, vol. 111, pp. 8788-8790, Jun. 2014. doi: 10.1073/pnas.1320040111.

[7] J. H. Fowler and N. A. Christakis,“Dynamic spread of happiness in a large socialnetwork: longitudinal analysis over 20 years in the framingham heart study,”BMJ , vol. 337, article no. a2338. doi: 10.1136/bmj.a2338.

[8] A. R. Damasio,“Descartes’error and the future of human life,”Scientific Ameri⁃can, vol. 271, no. 4, p. 144, Oct. 1994.

[9] R. W. Picard, Affective Computing. Cambridge, USA: The MIT Press, 1997, vol.167, p. 170.

[10] U. Dimberg,“Facial reactions to facial expressions,”Psychophysiology, vol. 19,no. 6, pp. 643-647, Nov. 1982. doi: 10.1111/j.1469⁃8986.1982.tb02516.x.

[11] P. Vuilleumier and G. Pourtois,“Distributed and interactive brain mechanismsduring emotion face perception: evidence from functional neuroimaging,”Neu⁃ropsychologia, vol. 45, no. 1, pp. 174-194, 2007. doi: 10.1016/j.neuropsycholo⁃gia.2006.04.0223.

[12] M. Eimer, and A. Holmes,“Event⁃related brain potential correlates of emotion⁃al face processing,”Neuropsychologia, vol. 45, no. 1, pp. 15- 31, 2007. doi:10.1016/j.neuropsychologia.2006.04.022.

[13] T. Flaisch,and H. T. Schupp,“Tracing the time course of emotion perception:the impact of stimulus physics and semantics on gesture processing,”SocialCognitive and Affective Neuroscience, vol. 8, no. 7, pp. 820-827, Jul. 2013. doi:10.1093/scan/nss073..

[14] E. C. Forscher, and W. Li,“Hemispheric asymmetry and visuo⁃olfactory inte⁃gration in perceiving subthreshold (micro) fearful expressions,”The Journal ofNeuroscience, vol. 32, no. 6, pp. 2159-2165, Feb. 2012. doi: 10.1523/JNEURO⁃SCI.5094⁃11.

[15] H. Kawasaki, N. Tsuchiya, C. K. Kovach, et al.,“Processing of facial emotionin the human fusiform gyrus,”Journal of Cognitive Neuroscience, vol. 24, no. 6,pp. 1358-1370, Jun. 2012.doi: 10.1162/jocn_a_00175.

[16] E. Redcay and T. A. Carlson,“Rapid neural discrimination of communicativegestures,”Social Cognitive and Affective Neuroscience, vol. 10, no. 4, pp. 545-551, Apr. 2015. doi: 10.1093/scan/nsu089.

[17] W. Li, R. E. Zinbarg, S. G. Boehm, and K. A. Paller,“Neural and behavioral ev⁃

idence for affective priming from unconsciously perceived emotional facial ex⁃pressions and the influence of trait anxiety,”Journal of Cognitive Neurosci⁃ence , vol. 20, no. 1, pp. 95-107, Jan. 2008. doi: 10.1162/jocn.2008.20006.

[18] S. Bentin, T. Allison, A. Puce, E. Perez, and G. McCarthy,“Electrophysiologi⁃cal studies of face perception in humans,”Journal of Cognitive Neuroscience,vol. 8, no. 6, pp. 551-565, Nov. 1996. doi: 10.1162/jocn.1996.8.6.551.

[19] E. Halgren, T. Raij, K. Marinkovic, V. Jousmaki, and R. Hari,“Cognitive re⁃sponse profile of the human fusiform face area as determined by MEG,”Cere⁃bral Cortex, vol. 10, no. 1, pp. 69-81, Jan. 2000. doi: doi.org/10.1093/cercor/10.1.69.

[20] M. J. Farah, K. D. Wilson, M. Drain, and J. N. Tanaka,“What is‘ special’about face perception?”Psychological Review vol. 105, no. 3, pp. 482- 498,Jul. 1998.

[21] C. J. Mondloch, T. L. Lewis, D. Robert, et al.,“Face perception during early in⁃fancy,”Psychological Science, vol. 10, no. 5, pp. 419- 422, Sept. 1999. doi:10.1111/1467⁃9280.00179.

[22] M. H. Johnson, S. Dziurawiec, H. Ellis, and J. Morton,“Newborns’preferentialtracking of face⁃like stimuli and its subsequent decline,”Cognition, vol. 40, no.1-2, pp. 1-19, Aug. 1991.

[23] Y. Sugita,“Face perception in monkeys reared with no exposure to faces,”Pro⁃ceedings of the National Academy of Sciences. vol. 105, no. 1, pp. 394- 398,2008. doi: 10.1073/pnas.0706079105.

[24] C. H. Hansen and R. D. Hansen,“Finding the face in the crowd: an anger supe⁃riority effect,”Journal of Personality and Social Psychology, vol. 54, no. 6, pp.917-924, Jul. 1988. doi: 10.1037/0022⁃3514.54.6.917.

[25] A. Öhman, D. Lundqvist, and F. Esteves,“The face in the crowd revisited: athreat advantage with schematic stimuli,”Journal of Personality and Social Psy⁃chology, vol. 80, no. 3, pp. 381- 396, Mar. 200). doi: 10.1037/0022 ⁃3514.80.3.381.

[26] E. Fox, V. Lester, R. Russo, et al.“Facial expressions of emotion: are angry fac⁃es detected more efficiently?”Cognition Emotion, vol. 14, no. 1, ppl. 61-92,Jan. 2000. doi: 10.1080/026999300378996.

[27] A. Öhman, P. Juth, and D. Lundqvist,“Finding the face in a crowd: Relation⁃ships between distractor redundancy, target emotion, and target gender,”Cogni⁃tion and Emotion, vol. 24, no. 7, pp. 1216-1228, 2010. doi: 10.1080/02699930903166882.

[28] E. Fox and L. Damjanovic,“The eyes are sufficient to produce a threat superiori⁃ty effect,”Emotion, vol. 6, no. 3, pp. 534-539, Aug. 2006. doi: 10.1037/1528⁃3542.6.3.534.

[29] M. G. Calvo and L. Nummenmaa,“Detection of emotional faces: salient physi⁃cal features guide effective visual search,”Journal of Experimental Psychology:General, vol. 137, no. 3, pp. 471-494, Aug. 2008. doi: 10.1037/a0012771.

[30] P. Juth, D. Lundqvist, A. Karlsson, and A. Öhman,“Looking for foes andfriends: perceptual and emotional factors when finding a face in the crowd,”Emotion, vol. 5, no. 4, pp. 379- 395, Dec. 2005. doi: 10.1037/1528 ⁃3542.5.4.379.

[31] P. Rozin and A. E. Fallon,“A perspective on disgust,”Psychological Review,vol. 94, no. 1, pp. 23-41, Jan. 1987.

[32] E. A. Krusemark and W. Li,“Do all threats work the same way? Divergent ef⁃fects of fear and disgust on sensory perception and attention,”The Jounal ofNeuroscience, vol. 31, no. 9, pp. 3429-3434, Mar. 2011. doi:10.1523/JNEURO⁃SCI.4394⁃10.2011.

[33] E. A. Krusemark and W. Li,“Enhanced olfactory sensory perception of threat inanxiety: an event⁃related fMRI study,”Chemosens Percept, vol. 5, no. 1, pp. 37-45, Mar. 2012. doi:10.1007/s12078⁃011⁃9111⁃7.

[34] Y. You andW. Li,“Parallel processing of general and specific threat during ear⁃ly stages of perception,”Social Cognitive Affective Neuroscience, vol. 11, no. 3,pp. 395-404, 2016. doi:10.1093/scan/nsv123.

[35] D. Sabatinelli, E. E. Fortune, Q. Li, et al.,“Emotional perception: meta⁃analy⁃ses of face and natural scene processing,”Neuroimage, vol. 54, no. 3, pp. 2524-2533, Feb. 2011. doi:10.1016/j.neuroimage.2010.10.011.

[36] P. Fusar⁃Poli, A. Placentino, F. Carletti, et al.,“Functional atlas of emotionalfaces processing: a voxel⁃based meta⁃analysis of 105 functional magnetic reso⁃nance imaging studies,”Journal of Psychiatry and Neuroscience, vol. 34, no. 6,pp. 418-432, Nov. 2009.

[37] L. Pessoa,“On the relationship between emotion and cognition,”Nature ReviewsNeuroscience, vol. 9, no. 2, pp. 148-158, Feb. 2008. doi:10.1038/nrn2317.

[38] D. A. Sauter, F. Eisner, A. J. Calder, and S. K. Scott,“Perceptual cues in non⁃verbal vocal expressions of emotion,”Quarterly Journal of Experimental Psy⁃chology (Hove), vol. 63, no. 11, pp. 2251- 2272, Nov. 2010. doi:10.1080/17470211003721642.

[39] E. T. Rolls, J. O'Doherty, M. L. Kringelbach, et al.,“Representations of pleas⁃

Special Topic



ant and painful touch in the human orbitofrontal and cingulate cortices,”Cere⁃bral Cortex, vol. 13, no. 3, pp. 308-317, Mar. 2003.

[40] C. McCabe, E. T. Rolls, A. Bilderbeck, and F. McGlone,“Cognitive influenceson the affective representation of touch and the sight of touch in the humanbrain,”Social Cognitive and Affective Neuroscience, vol. 3, no. 2, pp. 97-108,Jun. 2008. doi:10.1093/scan/nsn005.

[41] S. K. Scott, A. W. Young, A. J. Calder, et al.,“Impaired auditory recognition offear and anger following bilateral amygdala lesions,”Nature, vol. 385, no. 6613,pp. 254-257, Jan. 1997. doi:10.1038/385254a0.

[42] A. J. Calder, J. Keane, F. Manes, N. Antoun, and A. W. Young,“Impaired rec⁃ognition and experience of disgust following brain injury,”Nat Neurosci, vol. 3,pp. 1077-1078, 2000. doi:10.1038/80586.

[43] C. P. Said, J. V. Haxby, and A. Todorov,“Brain systems for assessing the affec⁃tive value of faces,”Philosophical Transactions of the Royal Society of London.Series B, Biological Sciences, vol. 366, no. 1572, pp. 1660-1670, Jun. 2011.doi:10.1098/rstb.2010.0351.

[44] J. Driver and T. Noesselt,“ Multisensory interplay reveals crossmodal influenc⁃es on‘sensory⁃specific’brain regions, neural responses, and judgments,’Neu⁃ron, vol. 57, no. 1, pp. 11-23, Jan. 2008. doi: 10.1016/j.neuron.2007.12.013.

[45] K. Vogeley and G. Bente,“‘Artificial humans’: psychology and neuroscienceperspectives on embodiment and nonverbal communication,’Neural Networks,vol. 23, no. 8 ⁃ 9, pp. 1077- 1090, Oct. ⁃ Nov. 2010. doi:10.1016/j.neu⁃net.2010.06.003 (2010).

[46] U. Frith and C. D. Frith,“Development and neurophysiology of mentalizing,”Philosophical Transactions of the Royal Society B⁃Biological Sciences, vol. 358,no. 1431, pp. 459-473, Mar. 2003. doi:DOI 10.1098/rstb.2002.1218.

[47] C. P. Said, C. D. Moore, A. D. Engell, A. Todorov, and J. V. Haxby,“Distribut⁃ed representations of dynamic facial expressions in the superior temporal sul⁃cus,”Journal of Vision, vol. 10, no. 5, p. 11, May 2010. doi:10.1167/10.5.11.

[48] K. S. LaBar, M. J. Crupain, J. T. Voyvodic, and G. McCarthy,“Dynamic percep⁃tion of facial affect and identity in the human brain,”Cerebral Cortex, vol. 13,no. 10, pp. 1023-1033, Oct. 2003.

[49] J. V. Haxby, E. A. Hoffman, and M. I. Gobbini,“Human neural systems forface recognition and social communication,”Biological Psychiatry. vol. 51, no.1, pp. 59-67, Jan. 2002.

[50] A. J. Calder and A. W. Young,“Understanding the recognition of facial identityand facial expression,”Nature Reviews. Neuroscience, vol. 6, no. 8, pp. 641-651, Aug. 2005. doi:10.1038/nrn1724 (2005).

[51] P. Belin, R. J. Zatorre, P. Lafaille, P. Ahad, and B. Pike,“Voice⁃selective areasin human auditory cortex,”Nature, vol. 403, pp. 309- 312, Jan. 2000. doi:10.1038/35002078.

[52] S. K. Scott, D. Sauter, and C. McGettigan,“Brain mechanisms for processingperceived emotional vocalizations in humans,”Handbook of Mammalian Vocal⁃ization: An Integrative Neuroscience Approach, vol. 19. pp. 187-197, 2010. doi:10.1016/B978⁃0⁃12⁃374593⁃4.00019⁃X.

[53] W. Li,“Learning to smell danger: acquired associative representation of threatin the olfactory cortex,”Frontiers in Behavioral Neuroscience. vol. 8, pp. 98,2014. doi:10.3389/fnbeh.2014.00098.

[54] N. M. Weinberger,“Associative representational plasticity in the auditory cor⁃tex: a synthesis of two disciplines,”Learning and Memory, vol. 14, no. 1⁃2, pp.1-16, Jan.⁃Feb. 2007. doi: 10.1101/lm.421807.

[55] A. Grosso, M. Cambiaghi, G. Concina, T. Sacco, and B. Sacchetti,“Auditorycortex involvement in emotional learning and memory,”Neuroscience, vol. 299,pp. 45-55, Jul. 2015. doi: 10.1016/j.neuroscience.2015.04.068.

[56] R. Tirindelli, M. Dibattista, S. Pifferi, and A. Menini,“From pheromones to be⁃havior,”Physiological Reviews. vol. 89, no. 3, pp. 921- 956, Jul. 2009. doi:10.1152/physrev.00037.2008.

[57] K. Touhara and L. B. Vosshall,“Sensing odorants and pheromones with chemo⁃sensory receptors,”Annual Review of Physiology, vol. 71, pp. 307-332, 2009.doi:10.1146/annurev.physiol.010908.163209.

[58] L. B. Kats and L. M. Dill,“The scent of death: chemosensory assessment of pre⁃dation risk by prey animals,”Ecoscience, vol. 5, no. 3, pp. 361-394, 1998.

[59] W. Turner,“The convolutions of the brain: a study in comparative anatomy,”Journal of Anatomy and Physiology, vol. 25, pp. 105-153, 1890.

[60] C. Bushdid, M. O. Magnasco, L. B. Vosshall, and A. Keller,“Humans can dis⁃criminate more than 1 trillion olfactory stimuli,”Science, vol. 343, no. 6177,pp. 1370-1372, Mar. 2014. doi:10.1126/science.1249168.

[61] M. Laska, A. Seibt, and A. Weber,“‘Microsmatic’primates revisited: olfactorysensitivity in the squirrel monkey,”Chemical Senses. vol. 25, no. 1, pp. 47-53,Feb. 2000. doi: 10.1093/chemse/25.1.47 .

[62] M. Laska, M. Fendt, A. Wieser, et al.,“Detecting danger⁃⁃or just another odor⁃ant? Olfactory sensitivity for the fox odor component 2,4,5⁃ trimethylthiazoline

in four species of mammals,”Physiology Behavior, vol. 84, no. 2, pp. 211-215,Feb. 2005. doi:10.1016/j.physbeh.2004.11.006.

[63] J. P. McGann,“Poor human olfaction is a 19th ⁃ century myth,”Science, vol.356, no. 6338, eaam7263, May 2017. doi: 10.1126/science.aam7263.

[64] S. S. Schiffman,“Physicochemical correlates of olfactory quality,”Science, vol.185, no. 4146, pp. 112-117, 1974. doi: 10.1126/science.185.4146.112.

[65] Y. Yeshurun and N. Sobel,“An odor is not worth a thousand words: from multi⁃dimensional odors to unidimensional odor objects,”Annual Review of Psycholo⁃gy, vol. 61, no. 219⁃241, 2010. doi: 10.1146/annurev.psych.60.110707.163639.

[66] S. T. Carmichael,M. C. Clugnet, and J. L. Price,“Central olfactory connectionsin the macaque monkey,”The Journal of Comparative Neurology, vol. 346, no.3, pp. 403-434, Aug. 1994. doi:10.1002/cne.903460306.

[67] J. A. Gottfried,“Central mechanisms of odour object perception,”Nature Re⁃views. Neuroscience, vol. 11, no. 9, pp. 628-641, Sept. 2010. doi: nrn2883 [pii]10.1038/nrn2883.

[68] W. Li, I. Moallem, K. A. Paller, and J. A. Gottfried,“Subliminal smells canguide social preferences,”Psychological Science, vol. 18, no. 12, pp. 1044-1049, 2007, doi: PSCI2023 [pii]10.1111/j.1467⁃9280.2007.02023.x.

[69] E. C. Forscher and W. Li,“Hemispheric asymmetry and visuo⁃olfactory integra⁃tion in perceiving subthreshold (micro) fearful expressions,”The Journal ofNeuroscience, vol. 32, no. 6, pp. 2159- 2165, 2012. doi: 10.1523/JNEURO⁃SCI.5094⁃11.2012.

[70] K. T. Lubke and B. M. Pause,“Always follow your nose: the functional signifi⁃cance of social chemosignals in human reproduction and survival,”Hormonesand Behavior, vol. 68C, pp. 134-144, Feb. 2015. doi: 10.1016/j.yhbeh.2014.10.001.

[71] S. Jacob, D. J. Hayreh, and M. K. McClintock,“Context ⁃dependent effects ofsteroid chemosignals on human physiology and mood,”Physiology & Behavior,vol. 74, no. 1⁃2, pp. 15-27, Sept. 2001.

[72] C. Ober,“Studies of HLA, fertility and mate choice in a human isolate,”Hu⁃man Reproduction Update, vol. 5, no. 2, pp. 103-107, Mar.⁃Apr. 1999.

[73] R. H. Porter and J. D. Moore,“Human kin recognition by olfactory cues,”Phys⁃iology & Behavior, vol. 27, no. 3, pp. 493-495, Sept. 1981. doi: 10.1016/ 0031⁃9384(81)90337⁃1.

[74] J. N. Lundstrom, J. A. Boyle, R. J. Zatorre, and M. Jones⁃Gotman,“The neuro⁃nal substrates of human olfactory based kin recognition,”Human Brain Map⁃ping, vol. 30, no. 8, pp. 2571-2580, Aug. 2009. doi:10.1002/hbm.20686.

[75] G. E. Weisfeld, T. Czilli, K. A. Phillips, J. A., Gall, and C. M. Lichtman,“Possi⁃ble olfaction ⁃ based mechanisms in human kin recognition and inbreedingavoidance,”Journal of Experimental Child Psychology, vol. 85, no. 3, pp. 279-295, Jul. 2003.

[76] H. Arakawa, S. Cruz, and T. Deak,“From models to mechanisms: odorant com⁃munication as a key determinant of social behavior in rodents during illness⁃as⁃sociated states,”Neuroscience and Biobehavioral Reviews, vol. 35, no. 9, pp.1916-1928, Oct. 2011. doi: 10.1016/j.neubiorev.2011.03.007.

[77] M. J. Olsson, J. N. Lundström, B. A. Kimball, et al.,“The scent of disease: hu⁃man body odor contains an early chemosensory cue of sickness,”PsychologicalScience, vol. 25, no. 3, pp. 817-823,Mar. 2014.doi: 10.1177/0956797613515681.

[78] L. R. Mujica⁃Parodi, H. H. Strey, B. Frederick, et al.,“Chemosensory cues toconspecific emotional stress activate amygdala in humans,”PLoS One, vol. 4,article no. e6415, 2009. doi: 10.1371/journal.pone.0006415.

[79] R. Zernecke, T. Frank, K. Haegler, et al.,“Correlation analyses of detectionthresholds of four different odorants,”Rhinology, vol. 49, no. 3, pp. 331-336,Aug. 2011. doi: 10.4193/Rhino10.263.

[80] B. M. Pause,“Processing of body odor signals by the human brain,”Chemosen⁃sory Perception, vol. 5, no. 1, pp. 55-63, Mar. 2012. doi: 10.1007/s12078⁃011⁃9108⁃2.

[81] A. Prehn⁃Kristensen, C. Wiesner, T. O. Bergmann, et al.,“Induction of empa⁃thy by the smell of anxiety,”PLoS One, vol. 4, article no. e5987, 2009. doi:10.1371/journal.pone.0005987.

[82] A. R. Radulescu and L. R. Mujica⁃Parodi,“Human gender differences in theperception of conspecific alarm chemosensory cues,”. PLoS One, vol. 8, articleno. e68485, 2013. doi: 10.1371/journal.pone.0068485.

[83] J. N. Lundstrom, J. A. Boyle, R. J. Zatorre, and M. Jones⁃Gotman,“Functionalneuronal processing of body odors differs from that of similar common odors,”Cerebral Cortex, vol. 18, no. 6, pp. 1466-1474, Jun. 2008. doi: 10.1093/cercor/bhm178.

[84] J. Driver and T. Noesselt,“Multisensory interplay reveals crossmodal influenc⁃es on‘sensory ⁃ specific’brain regions, neural responses, and judgments,”.Neuron, vol. 57, no. 1, pp. 11-23, Jan. 2008. doi: 10.1016/j.neuron.2007.12.013.

[85] M. M. Mesulam,“From sensation to cognition,”Brain, vol. 121, no. 6, pp.1013-1052, 1998.

Special Topic



[86] B. E. Stein and M. A. Meredith,“The Merging of the Senses. Cambridge, USA:The MIT Press, 1993.

[87] P. Maurage and S. Campanella,“Experimental and clinical usefulness of cross⁃modal paradigms in psychiatry: an illustration from emotional processing in al⁃cohol ⁃dependence,”Frontiers in Human Neuroscience, vol. 7, article no. 394,Jul. 2013. doi: 10.3389/fnhum.2013.00394.

[88] S. Campanella and P. Belin,“Integrating face and voice in person perception,”Trends in Cognitive Sciences, vol. 11, no. 12, pp. 535- 543, Dec. 2007. doi:10.1016/j.tics.2007.10.001.

[89] D. G. Amaral, J. L. Price, A. Pitkanen, and S. T. Carmichael,“Anatomical orga⁃nization of the primate amygdaloid complex,”in The Amygdala: Neurobiologi⁃cal Aspects of Emotion, Memory, and Mental Dysfunction. Newo York, USA:Wiley⁃Liss, 1992.

[90] T. Ethofer, S. Anders, M. Erb, et al.,“Impact of voice on emotional judgment offaces: an event⁃related fMRI study,”Human Brain Mapping, vol. 27, no. 9, pp.707-714, 2016. doi: 10.1002/hbm.20212.

[91] J. H. Kaas and C. E. Collins,“The resurrection of multisensory cortex in pri⁃mates: connections patterns that integrate modalities,”in The Handbook of Mul⁃tisensory Processes, G. A. Calvert, C. Spence, and B. E. Stein Eds. Cambridge,USA: MIT Press, 2004, pp. 285-293.

[92] M. Mesulam,“Defining neurocognitive networks in the BOLD new world ofcomputed connectivity,”Neuron, vol. 62, no. 1, pp. 1- 3, Apr. 2009. doi:10.1016/j.neuron.2009.04.001.

[93] M. T. Wallace, T. J. Jr. Perrault, W. D. Hairston, and B. E. Stein,“Visual expe⁃rience is necessary for the development of multisensory integration,”The Jour⁃nal of Neuroscience, vol. 24, no. 43, pp. 9580-9584, Oct. 2004. doi: 10.1523/JNEUROSCI.2535⁃04.2004.

[94] V. I. Muller, E. C. Cieslik, B. I. Turetsky, and S. B. Eickhoff,“Crossmodal in⁃teractions in audiovisual emotion processing,”Neuroimage, vol. 60, no. 1, pp.553-561, Mar. 2012. doi: 10.1016/j.neuroimage.2011.12.007.

[95] R. S. Herz,“The effect of verbal context on olfactory perception,”Journal ofExperimental Psychology. General, vol. 132, no. 4, pp. 595- 606, Dec. 2003.doi: 10.1037/0096⁃3445.132.4.595.

[96] M. L. Dematte, D. Sanabria, R. Sugarman, and C. Spence,“Cross⁃modal inter⁃actions between olfaction and touch,”Chemical Senses, vol. 31, no. 4, pp. 291-300, May 2016. doi: 10.1093/chemse/bjj031.

[97] J. Djordjevic, M. Jones⁃Gotman, K. De Sousa, and H. Chertkow,“Olfaction inpatients with mild cognitive impairment and Alzheimer’s disease,”Neurobiolo⁃gy of Aging, vol. 29, no. 5, pp. 693-706, May 2008. doi: S0197⁃4580(06)00437⁃4[pii]10.1016/j.neurobiolaging.2006.11.014.

[98] N. Mizutani, M. Okamoto, Y. Yamaguchi, et al.,“Package images modulate fla⁃vor perception for orange juice,”Food Quality and Preference, vol. 21, no. 7,pp. 867-872, Oct. 2010. doi: 10.1016/j.foodqual.2010.05.010.

[99] W. Li, R. E. Zinbarg, and K. A. Paller,“Trait anxiety modulates supraliminaland subliminal threat: brain potential evidence for early and late processing in⁃fluences,”Cognitive Affective & Behavioral Neuroscience, vol. 7, no. 1, pp. 25-36, Mar. 2007.

[100] W. Zhou and D. Chen,“Fear ⁃ related chemosignals modulate recognition offear in ambiguous facial expressions,”Psychological Science, vol. 20, no. 2,pp. 177-183, Feb. 2009. doi: 10.1111/j.1467⁃9280.2009.02263.x.

[101] J. M. Leppanen and J. K. Hietanen,“Affect and face perception: odors modu⁃late the recognition advantage of happy faces,”Emotion, vol. 3, no. 4, pp.315-326, Dec. 2003. doi:10.1037/1528⁃3542.3.4.315.

[102] W. Zhou and D. Chen,“Encoding human sexual chemosensory cues in the or⁃bitofrontal and fusiform cortices,”The Journal of Neuroscience, vol. 28, no.53, pp. 14416-14421, Dec. 2008. doi:10.1523/JNEUROSCI.3148⁃08.2008.

[103] J. Seubert, T. Kellermann, J. Loughhead, et al.,“Processing of disgusted facesis facilitated by odor primes: a functional MRI study,”Neuroimage, vol. 53,no. 2, pp. 746-756, Nov. 2010. doi: 10.1016/j.neuroimage.2010.07.012.

[104] L. R. Novak, D. R. Gitelman, B. Schuyler, and W. Li,“Olfactory⁃visual inte⁃gration facilitates perception of subthreshold negative emotion,”Neuropsycho⁃logia, vol. 77, pp. 288- 297, Oct. 2015. doi: 10.1016/j.neuropsycholo⁃gia.2015.09.005.

[105] S. Werner and U. Noppeney,“Distinct functional contributions of primary sen⁃sory and association areas to audiovisual integration in object categorization,”Journal of Neuroscience, vol. 30, no. 7, pp. 2662- 2675, Feb. 2010. doi:10.1523/JNEUROSCI.5091⁃09.2010.

[106] J. E. LeDoux,“Emotion: clues from the brain,”Annual Review of Psychology,vol. 46, pp. 209-235, Feb. 1995. doi: /10.1146/annurev.ps.46.020195.001233.

[107] R. Adolphs,“Recognizing emotion from facial expressions: psychological andneurological mechanisms,”Behavioral and Cognitive Neuroscience Reviews,vol. 1, no. 1, pp. 21-62, Mar. 2002.

[108] L. Pessoa and R. Adolphs,“Emotion processing and the amygdala: from a‘low road’to‘many roads’of evaluating biological significance,”Nature Re⁃view Neuroscience, vol. 11, no. 11, pp. 773- 783, Nov. 2010. doi:10.1038/nrn2920.

[109] A. T. Beck and D. A. Clark,“An information processing model of anxiety: au⁃tomatic and strategic processes,”Behaviour Research and Therapy, vol. 35,no. 1, pp. 49-58, Jan. 1997. doi: 10.1016/S0005⁃7967(96)00069⁃1.

[110] A. T. Beck, G. Emery, and R. Greenberg, Anxiety Disorders and Phobias: aCognitive Perspective. New York, USA: Basic, 1985.

[111] A. Ohman,“Fear and anxiety as emotional phenomena: clinical phenomenolo⁃gy, evolutionary perspectives, and information ⁃ processing mechanisms,”inHandbook of Emotions, M. Lewis and J. M. Haviland Eds. New York, USA:Guilford Press, 1993, pp. 511-536.

[112] A. Ohman,“Fear and anxiety: evolutionary, cognitive, and clinical perspec⁃tives,”in Handbook of Emotions (2nd ed.), M. Lewis & J. M. Haviland⁃Jones,Eds. New York, USA: The Guilford Press, 2000, pp. 573-593.

[113] V. Miskovic and A. Keil,“Acquired fears reflected in cortical sensory process⁃ing: a review of electrophysiological studies of human classical conditioning,”Psychophysiology, vol. 49, no. 9, pp. 1230- 1241, Sept. 2012. doi:10.1111/j.1469⁃8986.2012.01398.x.

[114] J. K. Olofsson and J. Polich,“Affective visual event⁃related potentials: arous⁃al, repetition, and time⁃on ⁃ task,”Biological Psychology, vol. 75, no. 1, pp.101-108, Jan. 2007. doi:10.1016/j.biopsycho.2006.12.006.

[115] E. C. Forscher, Y. Zheng, Z. Ke, J. Folstein, and W. Li,“Decomposing fearperception: a combination of psychophysics and neurometric modeling of fearperception,”Neuropsychologia, vol. 91, pp. 254- 261, Oct. 2016. doi:10.1016 /j.neuropsychologia.2016.08.018.

[116] S. J. Thorpe,“The speed of categorization in the human visual system,”Neu⁃ron, vol. 62, no. 2, pp. 168- 170, Apr. 2009. doi: 10.1016/j.neu⁃ron.2009.04.012.

[117] T. Brosch, G. Pourtois, and D. Sander,“The perception and categorisation ofemotional stimuli: a review,”Cognition Emotion, vol. 24, no. 3, pp. 377-400,2010. doi:Pii 91254516110.1080/02699930902975754.

[118] A. W. Young, D. Rowland, A. J. Calder, et al.,“Facial expression megamix:tests of dimensional and category accounts of emotion recognition,”Cogni⁃tion, vol. 63, no. 3, pp. 271-313, Jun. 1997.

[119] W. W. Seeley, V. Menon, A. F. Schatzberg, et al.,“Dissociable intrinsic con⁃nectivity networks for salience processing and executive control,”Journal ofNeuroscience, vol. 27, no. 9, pp. 2349-2356, Feb. 2007. doi:10.1523/JNEU⁃ROSCI.5587⁃06.2007.

[120] V. Menon and L. Q. Uddin,“Saliency, switching, attention and control: a net⁃work model of insula function,”Brain Structure and Function, vol. 214, no. 5⁃6, pp. 655-667, Jun. 2010.

[121] M. Corbetta and G. L. Shulman,“Control of goal⁃directed and stimulus⁃drivenattention in the brain,”Nature Reviews Neuroscience, vol. 3, no. 3, pp. 201-215, Mar. 2002.

[122] R. J. McNally,“Automaticity and the anxiety disorders,”Behaviour Researchand Therapy, vol. 33, no. 7, pp. 747-754, Sept. 1995. doi:10.1016/0005⁃7967(95)00015⁃P.

Manuscript received: 2017⁃08⁃02

LI Wen ([email protected]) is an associate professor of Psychology and Neurosci⁃ence and the director of the Cognitive Affective Neuroscience Laboratory, FloridaState University, USA. She received her Ph. D. in psychology from NorthwesternUniversity (USA) in 2004 and completed postdoctoral training in neuroscience atthe Medical School of Northwestern University in 2008. Her research centers on theinteraction between emotion and cognition and their implications in psychopatholo⁃gy. Dr. LI has won multiple awards including a Moskowitz Jacobs Inc. Award (equiv⁃alent to Young Investigator Award) from the Association for Chemoreception Scienc⁃es. She had received research support from the National Institute of Health (R01)and the Department of Defense. Dr. LI currently serves as a standing member of theCognition ad Perception Study Section for the National Institute of Health, USA.

BiographyBiography

Special Topic



Emotion Judgment System by EEG Based onEmotion Judgment System by EEG Based onConcept Base of EEG FeaturesConcept Base of EEG FeaturesMayo Morimoto1, Misako Imono2, Seiji Tsuchiya3, and Hirokazu Watabe3(1. Graduate School of Science and Engineering, Doshisha University, Kyo⁃Tanabe City 6100394, Japan;2. Daido University, Nagoya City 4578530, Japan;3. Doshisha University, Kyo⁃Tanabe City 6100394, Japan)

Abstract

This paper proposes an emotion judgment system by using an electroencephalogram (EEG) feature concept base with premise ofnoises included. This method references the word concept association system, which associates one word with other plural wordsand decides the relationship between several words. In this proposed emotion judgment system, the source EEG is input and 42EEG features are constructed by EEG data; the data are then calculated by spectrum analysis and normalization. All 2945 EEGdata of 4 emotions in the EEG data emotion knowledge base are calculated by the degree of association for getting the nearestEEG data from the EEG feature concept base constructed by 2844 concepts. From the experiment, the accuracy of the proposedsystem was 55.9%, which was higher than the support vector machine (SVM) method. As this result, the chain structured featureof the EEG feature concept base and the efficiency by the calculation of degree of association for EEG data help reduce the influ⁃ence of the noise.

electroencephalogram; EEG; emotion judgment; concept base; calculation of degree of associationKeywords

DOI: 10.3969/j. issn. 16735188. 2017. S2. 002http://kns.cnki.net/kcms/detail/34.1294.TN.20171214.1140.002.html, published online December 14, 2017

R1 Introduction

ecently, the studies of judging human’s emotionsby analyzing bio ⁃ information have been attractingattention and have become popular [1]. Emotionsare known to have strong relation to the amygdala

in the limbic system of the brain, and the relationship betweenemotions and the brain are being studied [2]. Emotions are im⁃manent acts, therefore, it is difficult to cognize hormonalchanges from external changes like expression or behavior. Forexample, sometimes people smiles to make other people notworry, even having“sad”emotion. Accordingly, judging an im⁃manent act is required for understating real emotion.

There are several measurement techniques for the brain,such as near infrared spectroscopy (NIRS), magnetoencepha⁃lography (MEG), and functional magnetic resonance imaging(fMRI). However, electroencephalogram (EEG) can measurewith simplified equipment and less burden to users. The EEGemotion judgment is expected to be widely used, for examplein medical care, in the near future. Owing to this, we need toconsider the scenes of usage in daily life for getting emotionfrom usual action. Although it is not a controlled environment

[2] for getting EEG for researches, and only using one type ofbio⁃information [3] thus far, it is eligible for getting EEG withsimplified equipment in daily life. However, it is difficult toprevent contamination of noise without a controlled environ⁃ment, which has an impact on the accuracy of EEG analysis.Some noises are possible to be rejected by signal processingtechnology, although it is difficult to reject all the noises com⁃pletely, even in a controlled environment.

Therefore, this paper proposes an emotion judgment systemby EEG with premise of noises included. This system is refer⁃ring to the word concept association system [4], proposed fornatural language processing. The word concept association sys⁃tem is constructed to realize word ⁃ to ⁃ word association andjudge the relevance between several words at the computer.This system is possible to get appropriate words even noisesare included. The word concept association system is construct⁃ed by a concept base and the calculation of degree of associa⁃tion. The concept base [5] is a large⁃scale database that is con⁃structed both manually and automatically using words frommultiple electronic dictionaries. The calculation of degree ofassociation [6] calculates the nearness of word meaning quanti⁃tatively, by using structure of the concept base. Natural lan⁃guage processing has a similar problem with word contamina⁃tion, but the calculation of degree of association surmounts theproblem by using the concept base to calculate efficiently.This work is partially supported by Japan Society for the Promotion ofScience (JSPS, 16K00311).

Special Topic


This method for quantifying the nearness between words in nat⁃ural language processing is reported to have better perfor⁃mance than the vector space method [7].

The proposed emotion judgment system in this paper is ro⁃bust in noise, using a concept base of EEG features and thecalculation of degree of association similar to that in the wordconcept association system. This proposed system replaces theword in word concept association system by EEG features.This paper compare the proposed system with the system usingthe support vector machine (SVM) [8], which known as highperformance for observing the efficacy.

2 Previous ResearchThere are various studies for using EEG to judge emotions,

such as hedonic judgment from stress in daily life [9], sensibili⁃ty evaluation by watching pictures [10], judgment of feeling byexperiment in past [11], and emotion judgment by listening mu⁃sic [12]. The hedonic judgment from stress judges comfort ordiscomfort, while the sensibility evaluation by watching pic⁃tures judges like or dislike. Especially, many studies in emo⁃tion recognition are defined emotion as a valence and an arous⁃al as a target of recognizing, which are difficult to understandwhich emotion is expressed intuitively [13]. These studies aresimilar by judging binary data. Furthermore, the judgment offeeling by experiment in past judges relax, pleasure, sadness,and anger, while the emotion judgment by listening music judg⁃es pleasure, sadness, anger, and happiness. The word“emo⁃tion”is used in all the studies, but its definition varies [14]-[19]. Although the research of emotion has been conducted fora long time in psychology and neuroscience, it is still difficultto define it exactly. Thus, a standard definition of emotion doesnot exist [19], [20]. The current research of engineering ap⁃proaches is also facing difficulty of defines“emotion”quantita⁃tively [21]. Therefore, this paper regards comfort/discomfort,feeling, excitement, concentration, and anxiety as affection,and pleasure, anger, sadness, and fear as emo⁃tions of high order than affection.

Similar to the emotions we define in this paper,there are two methods to quantify EEG: the emo⁃tion spectrum analysis and the emotion fractal⁃di⁃mension analysis [22], [23]. The emotion spec⁃trum analysis method proposed by Musha et al.[22] judges such emotions as anger/stress, plea⁃sure, sadness, and relax by using EEG. The emo⁃tion fractal⁃dimension analysis proposed by Nak⁃agawa [2] uses EEG to judge anger, pleasure, sad⁃ness, and relax. These methods use the datawhich EEG and emotions are paired. These dataare acquired by test subjects, which trained howto imaging emotions to control them. This paperassumes to use the system in daily life, thus thedata are acquired from non ⁃ trained test subjects

by showing them one Japanese film.Additionally, EEG is one kind of bio⁃information and noises

are easily included. It is difficult to remove all noises complete⁃ly even by trying policy of noise elimination [24], [25]. There⁃fore, the proposed system in this paper reduces the influenceof noises, presupposing that noises are included rather than re⁃ducing noises themselves.

3 Overview of Proposed SystemThe objective of this proposed system is to obtain the emo⁃

tions of conversation partners by reading their EEGs. Fig. 1outlines the proposed emotion judgment system by EEG.

EEGs acquired from the test subjects are used as sourceEEGs. Emotions of the subject are acquired simultaneously assource EEGs. Emotions are assigned to spectrum analysis ofthe source EEGs, which is performed every 1.28 second. TheEEG features are determined to θ waves (4.0 Hz to 8.0 Hz), αwaves (8.0 Hz to 13.0 Hz), and β waves (13.0 Hz to 30.0Hz),as shown in Fig. 2.

Emotions are judged from the EEG features by an associa⁃tion mechanism. The association of EEG Features is realizedby using a huge concept base, which is automatically builtfrom the EEG Features. The relationship between the EEGFeatures is evaluated by calculating the degree of association.The concept base and the degree of association are generallyused for natural language processing; because the structure ofEEG Features and that of the word in natural language process⁃ing are similar, we apply the technique in our system. Hereaf⁃ter, this concept base and the calculation method are calledthe association mechanism. Emotions judged in the proposedsystem include pleasure, anger, sadness, and fear.3.1 Source EEG and Emotion

The EEGs were measured at 14 locations that conform to theInternational 10/20 System [26] (Fig. 3).

EEG: electroencephalogram KB: knowledge base

Output(Emotion)

▲Figure 1. Emotion Judgment System by EEG.

Spectrumanalysis

Normalization(linear and non⁃linear)

EEG dataemotion

KB

JudgingemotionfromEEG

Input(EEG)

EEG featureconceptbase

Associationmechanism

Emotion judgment system by EEG

Special Topic

Emotion Judgment System by EEG Based on Concept Base of EEG FeaturesMayo Morimoto, Misako Imono, Seiji Tsuchiya, and Hirokazu Watabe


Calculation of degree of association

The test subjects were fitted with electroencephalographycaps [27] and asked to watch a Japanese film for approximatelytwo hours. During the film watching, the subjects were askedto gauge the emotions they had felt by the speeches in the film,and the source EEGs were acquired simultaneously. The scenefor a speech in the film was frozen, and the subjects wereasked what emotions they felt at the time of watching the scene.

20 subjects (10 males and 10 females) were used, and theviewing was divided into four sessions to reduce the physicalburden of the subjects. Before and after the film, the EEGs ofopen⁃eye and closed⁃eye states were respectively measured for

approximately one minute for the normalization ofEEG features.3.2 EEG Features and EEG Data

The source EEGs are assigned to spectrum anal⁃ysis and performed every 1.28 second. The EEGfeatures are determined to take average of eachwave: θ waves (4.0 Hz to 8.0 Hz), α waves (8.0 Hzto 13.0 Hz), and β waves (13.0 Hz to 30.0 Hz).The EEG features are made by average of 14 loca⁃tions and 3 frequency bands, therefore, 42 EEGfeatures are calculated from 1.28⁃second EEG.3.3 Normalization of EEG Features

The EEGs show changes in voltage intensityover time within an individual, and the base volt⁃age intensity differs among individuals. For this

reason, the possibility of misjudgment exists because those val⁃ues differ greatly even among EEGs with similar waveforms.To solve this problem, linear normalization and non⁃linear nor⁃malization were performed.3.3.1 Linear Normalization

The linear normalization was performed because voltage in⁃tensity of EEGs varies over time depending on the subject.Since the eyes were open while viewing the film, linear normal⁃ization [28] was performed to acquire EEGs both before and af⁃ter the experiment based on EEG features from the eye ⁃openstate. EEG features Linear_alij was obtained by linear norma⁃lization of the first EEG feature alij at a certain point of timeduring the experiment, and is expressed as:

where j HZ frequency is taken from the i⁃th location in electro⁃encephalography caps and l blocks from the experiment start.The blocks are calculated by time that is divided by 1.28 sec⁃onds. P1 is the time of the experiment start, and p2 is the timeof the experiment end. q1 is the voltage when the experimentstarted, and q2 is that when it ended.3.3.2 Non⁃Linear Normalization

The non⁃linear normalization was performed to take into ac⁃count the difference among individuals in base voltage intensi⁃ty. The non ⁃ linear normalized values were obtained by using(2). f ( )x is the EEG features after the non ⁃ linear normaliz⁃ation [28], x is the EEG features applied in the non⁃linear nor⁃malization, xmin is the minimum EEG features of an individual,and xmax is his maximum EEG features. As a result, the EEGfeatures with large values are compressed and EEG featureswith small value are expanded by the non⁃liner normalization.Thus, the degree of voltage intensity of an individual’s EEGs

▲Figure 2. Spectrum analysis of the source EEGs.

C: Central F: Frontal O: Occipital P: Parietal T: Temporal

▲Figure 3. EEG measurement at 14 locations, conforming to theInternational 10/20 System.

Fp1 Fp2

F3 F4Fz

T3 C3 C4 T4

P3 P4Pz

O1 O2

28.79332.000

25.59422.39519.19515.99612.7979.5986.3983.199

Freque

ncy(Hz

)

27.124.121.118.015.012.09.06.03.0Voltage (μV)

Emotion Judgment System by EEG Based on Concept Base of EEG FeaturesMayo Morimoto, Misako Imono, Seiji Tsuchiya, and Hirokazu Watabe

Special Topic


Linear_alij = alij + ìíî

üýþ

æèç

öø÷

q1 - q2p2 - p1 × l + q2 -

æèç

öø÷

q2 - q1p2 - p1 × l + q2 /2, (1)

is solved.

f ( )x = log( )x - xminlog( )xmax - xmin . (2)

3.4 EEG Data Emotion Knowledge BaseThe EEG data emotion knowledge base is a database con⁃

structed by EEGs and emotions. EEG features of 42 represent⁃ed by three bandwidths are obtained from 14 locations and as⁃sumed to be one EEG datum, and the EEG data are matchedwith emotions, either anger, sadness, fear, or pleasure. TheEEG data emotion knowledge base is consisted with the knowl⁃edge base of each emotion, and contains 2945 EEGs obtainedby excluding the outliers and noises. The emotions of the 2945EEGs are comprised 629 anger features, 857 sadness features,979 fear features, and 480 pleasure features.

4 Association MechanismThe association mechanism [5] consists of the concept base

[6] and the degree of association [7]. The concept base gener⁃ates semantics from certain EEG features. The degree of associ⁃ation is used for the semantics expansion, and it expresses therelationship between EEG features by a numeric value. Themethods of a concept base and the degree of association wereoriginally proposed for natural language processing, and are ap⁃plied to the EEGs in this paper.4.1 EEG Features Concept Base

The concept base [6] is originally a large⁃scale database thatis constructed both manually and automatically using wordsfrom multiple electronic dictionaries. The entry words in dic⁃tionaries used as concepts and the independent words in theexplanation under the concept are used as attributes. In thecurrent research, a concept base contains approximately 90,000 concepts, in which auto ⁃ refining processing is conductedafter the base is manually constructed. In this process, attri⁃butes are considered from the point of view of human sensibili⁃ty; the inappropriate attributes are deleted and necessary attri⁃butes are added.

In the concept base, Concept A is expressed by Attribute ai,indicating the features and the meaning of the concept in rela⁃tion to a Weight wi, denoting how important Attribute ai is inexpressing the meaning of Concept A. Assuming that the num⁃ber of attributes of Concept A is N, Concept A isexpressed by (3), where Attribute ai is called Pri⁃mary Attribute of Concept A.

A ={ }( )a1,w1 ,( )a2,w2 ,…,( )aN,wN . (3)Because Primary Attribute ai of Concept A is

defined as the concepts in the concept base, at⁃tributes can be similarly elucidated from ai. TheAttributes aij of ai are called Second Attributes

of Concept A. Attribute ai is defined by aij, and also defined asthe concepts, so aij is Primary Attribute of ai. Thus, the conceptbase can be connected to N⁃dimension.

In our work, the concept base is made by source EEGs in⁃stead of electronic dictionaries. In fact, EEG features are usedinstead of words. A word notation is used as a concept and anattribute in

AnInternationalICTR&DJournalSponsoredbyZTECorporation ... · ISSN1673-5188 CN34-1294/TN CODENZCTOAK...

Documents

Transcript of AnInternationalICTR&DJournalSponsoredbyZTECorporation ... · ISSN1673-5188 CN34-1294/TN CODENZCTOAK...