Emotional awareness in autonomous driving...Emotional awareness in autonomous driving Challenges,...
Transcript of Emotional awareness in autonomous driving...Emotional awareness in autonomous driving Challenges,...
Emotional awareness in autonomous driving
Challenges, Approaches & Vision
Luis Gressenbuch, Sebastian Bergemann
Technical University of Munich
Department of Informatics
[email protected], [email protected]
Seminar Emotional awareness in autonomous driving SS2019
Munich, Jun. 28th 2019
Outline
2Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
IntroductionEmotion
recognition
Emotion-aware
applicationsin AD
Challenges Conclusion
Memory
Attention
Problem solving
Drivingpleasure
Decisionmaking
Motivation
3Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Impacts of emotions on the driving task
Source: Affectiva
Safety Comfort
Motivation
4Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
What are the goals of emotional awareness in autonomous driving?
Emotion classification
5Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Basic Emotions (Ekman) Circumplex Model (Russell & Barrett)
[1] [2]
• Relationship between arousal and cognitive
performance
• Low arousal results in sleepiness
• High arousal results in stress
• Optimal level depends on the difficulty of the
task
The Yerkes-Dodson law
6Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[3]
IntroductionEmotion
recognition
Emotion-aware
applications in AD
Challenges Conclusion
Outline
7Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
A. Facial expressions
B. Thermal imaging
C. Physiological signals
D. Driving behavior
E. Speech
F. Gesture, head pose & eye gaze
G. Multimodal fusion
Approaches
8Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Emotion recognition – A pattern recognition task
9Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
ConventionalPre-
processingSegmentation
Feature extraction
Classification Tracking
Deep Learning
• Six basic emotions
• Facial Action Coding System (FACS)
▪ 28 Action Units
▪ Based on the anatomy of facial muscle groups
▪ Action Units can be present with different
intensity
▪ Emotions correlate with combinations of Action
Units
• Measurement:
▪ Electromyography
▪ Camera
Approaches – A) Facial expressions
10Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[4]
Features
Geometricfeatures
Spatial
Temporal
Appearancefeatures
Global
Local
Approaches – A) Facial expressions
11Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[5]
[6] [7]
[7]
• Deep learning derives features from training data
• Visualization of learned features showed
similarities to Action Units
+ Features do not need to be constructed manually
+ Better accuracy
- Time consuming optimization of hyperparameters
- Needs higher amount of computational resources
Approaches – A) Facial expressions
12Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Deep Learning
[8]
• Illumination
➢ Near-infrared cameras
• Head pose
➢ Stereo cameras
• Occlusion
• Mock expressions
• Emotions sharing the same expression
Approaches – A) Facial expressions
13Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Challenges
[9]
[9][9]
• Local temperature from arousal or muscle activity
• Similar methods as in visual imaging
• Facial Thermal Feature Points (FTFPs)
+ Robust towards illumination
+ Facilitated segmentation
+ Resistant against manipulation
- Thermodynamic effects
- High cost
Approaches – B) Thermal imaging
14Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[10]
Source: Flir
• Signals:
▪ Electrocardiography (ECG) /
Photoplethysmogram (PPG)
▪ Electrodermal activity (EDA)
▪ Skin temperature (SKT)
• Single signal can indicate arousal
• Multiple differential signals can indicate basic
emotions
Approaches – C) Physiological signals
15Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Autonomic Nervous System
[11]
• Electroencephalography (EEG)
• Measures local electrical activity of the brain
• Most common feature: Power spectral density
at different frequency bands
• Circumplex model
Approaches – C) Physiological signals
16Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Central Nervous System
[12]
Theta Alpha Beta Gamma
Arousal
Valence
• Need contact-based sensor equipment
➢ Smart wearables
➢ Highly integrated sensors
• Signal quality
Approaches – C) Physiological signals
17Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Challenges
[13]
• Distracted drivers
▪ drive slower
▪ steer less frequently
▪ steer with higher angles
• Signals
▪ Vehicle speed
▪ Steering wheel angle
▪ Throttle position
▪ Lateral deviation from the lane
▪ Relative vehicle kinematics
+ Availability of signals from CAN-Bus
- Indirect measurement of emotions
- Driving behaviour not available when control is automated
Approaches – D) Driving behavior
18Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[14]
Approaches – E) Speech
20Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[15]
ASR systems
Linguistic way:
• Speech recognition systems
• Databases with keywords linked to emotions
▪ Match all words to hyper-classes
▪ Match most beneficial words to specific emotions
• Challenges:
▪ False word detection
▪ Context complexity
Approaches – E) Speech
21Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[15]
Non-Linguistic way:
• Features:
Approaches – E) Speech
22Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Feature categories
Low-level descriptors (LLDs)
Prosodic
speed rate
pitch
pause
Spectral
amplitude
MFCCs
short energy
Functionals
extreme values
means
offset
[15]
Non-Linguistic way:
• Challenges:
▪ Differences (language, culture, age, gender, etc.)
▪ Acted comparison material
▪ Signal-Noise-Ratio of audio recordings
▪ Not available all the time
Approaches – E) Speech
23Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[15]
Approaches – F) Gesture, head pose & eye gaze
24Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[16]
• Detection:
▪ Wearable sensors (accelerometers or body markers)
▪ Cameras and computer vision
• Features:
▪ Raw features (3D data points)
▪ Velocity, acceleration and fluidity
▪ Quantity of motions (QoM)
▪ Contraction index (CI)
▪ Specific movements/positions like PERCLOS, etc.
Approaches – F) Gesture, head pose & eye gaze
25Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[16]
• Challenges:
▪ Car interior limitations (movement space)
▪ Few body expressions – broader emotion dimensions
▪ Either intrusive (markers) or more noisy (without them)
➢ Not a stand-alone but a support modality
Approaches – F) Gesture, head pose & eye gaze
26Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[16]
Approaches – G) Multimodal fusion
27Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[17]
One modality:
• Sensitive to noisy conditions, occlusion, etc.
• Can fail completely (recording problem)
Multiple modalities:
• Redundancy
• Diversity
➢ Increasing accuracy and robustness
Fusion variants:
• On feature-level
• On decision-level
Approaches – G) Multimodal fusion
28Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
[17]
IntroductionEmotion
recognition
Emotion-aware
applications in AD
Challenges Conclusion
Outline
30Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
A. Adapting driving related behavior
B. Improving the driver state
C. Adapting the HMI
• Distractions are a major reason for accidents
• Level 2: Driver constantly needs to monitor the vehicle
• Level 3: Distractions can lead to a loss of
situational awareness
➢ Increased takeover times
• Modalities:
▪ Eye gaze
▪ Head pose
▪ Facial expressions
▪ Driving behavior
• Proposed application: Lane Departure Warning (LDW)
Applications – A) Adapting driving related systems
32Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Distraction detection
Distraction
Cognitive
Visual Manual
• Driver state adaptive forward collision warning
(FCW) system
• System only warns if 𝑑𝑤 < 𝑑𝑟𝑒𝑙• Result: +10% accuracy, +40% precision
• Benefits:
+ Less false positives
+ Increased safety
Applications – A) Adapting driving related systems
33Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Reaction-time estimation on cognitive workload
[18]
[17]
• In a study drivers preferred second to the fastest,
routes that impose the least stress
• How to identify stressing routes?
➢ Database of geo-tagged heart rate variability
recordings of drivers
➢ Creates a heat map
Applications – A) Adapting driving related systems
34Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Routing
[19]
• “Happy drivers are better drivers”
• Recognizing negative emotions:
➢ Influencing the driver‘s emotional state so that
it becomes more positive
Applications – B) Improving the driver state
36Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
In general
[20]
1. Driver state display
Emotional feedback
2. Voice-based HMI
Calming down and objective reasoning
3. Subliminal influencing
Overall atmosphere (temperature, light, music)
Applications – B) Improving the driver state
37Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Proposed strategies
[20]
• 1,940 accidents with personal injury due to drowsy
driving in 2017 in Germany
• Drowsiness can be induced by driving autonomously
for a long time period
• Detection can support takeover management and
notification / suggestion system
Applications – B) Improving the driver state
38Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Drowsiness detection
Source: Daimler Global Media Site
• Driver assistance to increase comfort and safety
• Natural communication (contextual)
➢ Driver recognizes digital emotions
• Inconsistent pairing of emotions can be a safety
hazard
➢ VICO has to be empathic
Applications – C) Adapting HMI
39Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Virtual co-driver (VICO)
Source: BMW
• Notifications can critically increase driver‘s stress
level in complex situations
• Suppressing not immediately relevant notifications
based on current stress level
• Supported by improvement predictions
➢ Reduce annoyance and additional stress
Applications – C) Adapting HMI
40Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Stress-adaptive notifications
Source: BMW
IntroductionEmotion
recognition
Emotion-aware
applications in AD
Challenges Conclusion
Outline
41Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Research
• Limited comparison possibility
• Many applications in pilot status
• Insufficient field studies (regarding reliability)
Emotions
• Emotion definition and recognition is difficult even for humans
• Emotion differences based on context, culture, age, gender, etc.
Applications
• User acceptance unclear
• Constant monitoring
• Privacy concerns
• Ethical concerns
General challenges
42Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
IntroductionEmotion
recognition
Emotion-aware
applications in AD
Challenges Conclusion
Outline
43Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
• Many approaches for emotion recognition
➢ specific limitations for autonomous vehicle field
• Applications for autonomous driving are available
➢ mainly in early stage
➢ currently refer more to ADAS than to autonomous vehicles
• Challenges exist
➢ first solutions have been published
Conclusion
44Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision
Source: Affectiva
More autonomous
vehicle research
More emotion awareness
research in this field
[1] O. Spindler, “Affective space interfaces,” Diplomarbeit, TU Wien, Wien, 2009.
[2] J. A. Russell and L. F. Barrett, “Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant,” J. Pers. Soc. Psychol., vol. 76, no. 5, pp. 805–819, May 1999.
[3] A. Saeed, S. Trajanovski, M. van Keulen, and J. van Erp, “Deep Physiological Arousal Detection in a Driving Simulator Using Wearable Sensors,” in 2017 IEEE International Conference on Data Mining Workshops
(ICDMW), New Orleans, LA, 2017, pp. 486–493.
[4] E. Vural, M. Çetin, A. Erçil, G. Littlewort, M. Bartlett, and J. Movellan, “Machine Learning Systems for Detecting Driver Drowsiness,” in In-Vehicle Corpus and Signal Processing for Driver Behavior, K. Takeda, H.
Erdogan, J. H. L. Hansen, and H. Abut, Eds. Boston, MA: Springer US, 2009, pp. 97–110.
[5] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild,” in 2016 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5562–5570.
[6] D. Ghimire and J. Lee, “Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines,” Sensors, vol. 13, no. 6, pp. 7714–7734, Jun. 2013.
[7] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on Local Binary Patterns: A comprehensive study,” Image Vis. Comput., vol. 27, no. 6, pp. 803–816, May 2009.
[8] R. Breuer and R. Kimmel, “A Deep Learning Perspective on the Origin of Facial Expressions,” ArXiv170501842 Cs, May 2017.
[9] A. Yüce, H. Gao, G. L. Cuendet, and J. Thiran, “Action Units and Their Cross-Correlations for Prediction of Cognitive Load during Driving,” IEEE Trans. Affect. Comput., vol. 8, no. 2, pp. 161–175, Apr. 2017.
[10] C. Puri, L. Olson, I. Pavlidis, J. Levine, and J. Starren, “StressCam: Non-contact Measurement of Users’ Emotional States Through Thermal Imaging,” in CHI ’05 Extended Abstracts on Human Factors in Computing
Systems, New York, NY, USA, 2005, pp. 1725–1728.
[11] T. K. L. Hui and R. S. Sherratt, “Coverage of Emotion Recognition for Common Wearable Biosensors,” Biosensors, vol. 8, no. 2, Mar. 2018.
[12] S. Koelstra et al., “DEAP: A Database for Emotion Analysis ;Using Physiological Signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31, Jan. 2012.
[13] A. Riener, M. Jeon, I. Alvarez, and A. K. Frison, “Driver in the Loop: Best Practices in Automotive Sensing and Feedback Mechanisms,” in Automotive User Interfaces: Creating Interactive Experiences in the Car, G.
Meixner and C. Müller, Eds. Cham: Springer International Publishing, 2017, pp. 295–323.
[14] S. Choi, J. Kim, D. Kwak, P. Angkititrakul, and J. Hansen, “Analysis and Classification of Driver Behavior Using in-Vehicle CAN-BUS Information,” Bienn Workshop DSP -Veh Mob Syst, Jan. 2007.
[15] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, “Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,” Artif. Intell. Rev., vol. 43, no. 2, pp. 155–177, Feb. 2015.
[16] H. A. Vu, Y. Yamazaki, F. Dong, and K. Hirota, “Emotion recognition based on human gesture and speech information using RT middleware,” in 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE
2011), 2011, pp. 787–791.
[17] L. Kessous, G. Castellano, and G. Caridakis, “Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis,” J. Multimodal User Interfaces, vol. 3, no. 1,
pp. 33–48, Mar. 2010.
[18] V. Govindarajan, K. Driggs-Campbell, and R. Bajcsy, “Affective Driver State Monitoring for Personalized, Adaptive ADAS,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018,
pp. 1017–1022.
[19] S. D. Nadai et al., “Enhancing safety of transport by road by on-line monitoring of driver emotions,” in 2016 11th System of Systems Engineering Conference (SoSE), 2016, pp. 1–4.
[20] M. Braun, J. Schubert, B. Pfleging, and F. Alt, “Improving Driver Emotions with Affective Strategies,” Multimodal Technol. Interact., vol. 3, no. 1, p. 21, Mar. 2019
References
45Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision