Human-Robot Emotional and Musical Interactions...Human-Robot Emotional and Musical Interactions...
Transcript of Human-Robot Emotional and Musical Interactions...Human-Robot Emotional and Musical Interactions...
Human-Robot Emotional and Musical
Interactions Massimiliano Zecca
Professor of Healthcare Technology [email protected]
Special lecture for the International Journal of Advanced Robotic Systems
Outline of the presentation
• Emotional robotics • Musical robotics • Wearable bioinstrumentation
The pyramid of life
Increasing population Stable population Decreasing population
The pyramid of life is becoming more and more unstable
http://www.ipss.go.jp/index-e.html
1930 – the past
Over 65 <5%
2005 – the present
Over 65 about 20%
2050 – the future
More than 30% of the population in Japan will be over 65 years old!!!
Seamless mobility
Natural communication
Safety technology
Pseudo-lite
GPS
Environmental information structuring
Pseudo-lite (outdoor)
Autonomous environment recognition RFID tag
RT furniture
Interaction with physical environment
Position identification
Robot as physical agent
Emotional robot WE-4RII
Paul Ekman’s classification of emotions
• Ekman’s list of basic emotions (1970’): • anger • fear • surprise
ANGER DISGUST FEAR HAPPINESS SURPRISE SADNESS
The facial expressions of the 6 primary emotions are common to all cultures
FACS (Facial Action Coding System)
• Face elements
1. Miwa’s model 2. Kismet’s
model 3. Circumplex
model 4. Plutchick’s
Wheel of Emotions
1 2
3 4
Emotional models
Previous Studies
Tactile and Audition Olfaction
Motion Perception Depth Perception Light Perception Facial Expression
WE-2 (1995) WE-3 (1996) WE-3R (1997) WE-3RII (1998)
WE-3RIII (1999) WE-3RIV (2000) WE-3RV (2001)
Facial Color
WE-4 (2002)
New Model
External Environment
Internal Environment
Autonomic Reflex
Robot
Sensing
Sensing Personality
Emotion Need
Motion
Motion
Expression Personality
Emotion Need
Emotion Mood
Need
Consciousness
Mental Model for Humanoid Robot
Motion Reflex Sensing
Recognition Behavior Intelligence
WE-4RII emotional expressions
WE-4RII Various behaviors
WE-4RII consciousness
Big issues (1)
The robot works, but… • What about the people interacting with the robot? • Is s/he enjoying the interaction? • What about her/his emotions?
Musical Robots WF-4 and WAS-3
Interacting with humans at musical level
Long term research objective
natural and intuitive human-robot complete musical interaction
Waseda Flutist Robot N.4 Refined VI
WF-4RVI (2010)
1600
[mm
]
Lungs
Lips
Arms
Neck Fingers
Eyes
Lungs 2-DOF
Vibrato 1-DOF
Eyes 3-DOF
Arms 14-DOF
Lips 3-DOF
Fingers 12-DOF
Neck 4-DOF
Tonguing 1-DOF
DOF Configurations
Total : 41‐DOF X
Z
Y
Waseda Flutist Robot N.4 Refined VI
WAseda Saxophonist No.3
WAS-3
1400
[mm
]
x
z y
Oral cavity/1-DOF Tongue /1-DOF
Pump / 1-DOF
Valve / 1-DOF
y x
z
Lips/2-DOF
Air Flow
x z y
Eyes/2-DOF
Right/8-DOF
179[
mm
]
Left/11-DOF
185[
mm
]
Lung
Waist /1-DOF
Hands/19-DOF
計/28-DOF
WAseda Saxophonist
Big issues (2)
• What about the people interacting with the robot? • Is s/he enjoying the interaction? • What about her/his emotions?
• What about the interaction with other musicianso and artists in a small group?
• And hat about the interaction within a big orchestra?
Wearable sensing Common need: Understand the human being
Main Scientific interest
• Observation, analysis, and understanding of the human being • extreme and exquisite example of robotic system, • clarification of the basic mechanisms underlying
human's control of their bodies • extremely important and helpful tool for the society for
realizing better health-care systems, human-support devices, teleoperation methods, and so on.
Main Research Activities
• OBJECTIVE: • to develop a Wearable Bio-
instrumentation System capable of monitoring and integrating the human motion with several physiological indices
• Core research tool for understanding the basic mechanisms underlying human’s control of their bodies
• Ultimate objective: • to improve and enhance the quality of life of elderly and disabled people
House & Health care
Wearable Bio-
instrumentation System
Research approach and activities
Research approach and activities
• WB-4 and WB-4R Inertial Measurement Units
• WB-EMG wireless sensor
Industrial cooperation → WB Evolution
WB-1
2004
WB-1R
2005 2006
WB-2 WB-2R
2007
WB-3
2008
WB-4
2010
Z X
Y
Z X
Y
Z X
Y
Z X
Y
WB-4 WB-4
Best Paper Finalist Award, ROBIO 2007 Best Paper Award, AIM 2009 Best Paper Finalist Award, IRIS 2010
Best Presentation Award, WWLS 2010 Best Paper Finalist Award, ROBIO2010 Best Paper Award, ROBIO2012
2014
?WB-5?
12mm
WB-4 Wireless IMU (7g)
Bluetooth
Mother board
WB-4 Wireless IMU
q Bluetooth module: Zeal-‐C01 (Musenka Company) q Li-‐Polymer ba3ery : KPL501717 (3.6V@80mAh, Kayo Inc.) q WB-‐4 IMU power consump>on: 3.0V@80mA q Con>nuous working >me: 1 hour q Data: ±8 [G], ±1600 [deg/s] @200Hz
WB-EMG Sensor Specifications
Circuits WB-‐EMG Board Gain 2800
PCB Size (mm) 30.5x17.5x11mm Voltage Supply 3.7V (Poly Li-‐ion baRery) Sampling Rate 1KHz ResoluUon 12-‐bit
Low Frequency Cutoff 19Hz
High Frequency Cutoff 457Hz
Transmission Frequency 2.4 GHz (Bluetooth)
Power line Noise RejecUon Method
Amplifier Common-‐Mode RejecUon Driven Right Leg
DC Input RejecUon Passive DifferenUal Filter
DC Offset Removal from INA Feedback Integrator
Movement ArUfact RejecUon Input Filter, Feedback Integrator
IC Packages 8-‐MSOP, 14-‐TSSOP
Passive Components 16
AcUve Components 2
Components Count 18
ICMA2013, best student paper finalist
Research approach and activities
• Orientation estimation • Wavelet Based EMG
Denoising • EMG-driven muscle model
Quaternion-based Extended Kalman Filter
1)( −+= kTkkk
Tkkk RHPHHPK
Effect of R-Adaptive algorithm
ü Perfect bias canceling No dynamic error rejecUon ü Dynamic error rejecUon Drie
ü Dinamic error RejecUon ü No Drift
Denoising: Baseline Adaptive Denoise Algorithm (BADA)
Successes Limita>ons ü ReducUon of false acUvaUons 92% vs original
signal and 10% vs Standard Dohono Technique
Ø Heavy ComputaUonal Load of Wavelet Algorithm -‐> difficult to implement Real Time
ü No negaUve impact the signal original shape Ø Firmware currently implemented only on MATLAB
L. Bartolomeo, M. Zecca, et al, “Wavelet thresholding technique for sEMG denoising by baseline estimation” in International Journal of Computer Aided Engineering and Technology, vol. 4, no. 6, p. 517, 2012 L. Bartolomeo, M. Zecca, et Al., “Baseline Adaptive Wavelet Thresholding Technique for sEMG Denoising”, AIP Conference Proceedings Vol. 1371, ISSN Print+Online: 0094-243X, ISSN Online only: 1551-7616, ISBN: 978-0-7354-0931-6, 2011
Proposed EMG+IMU Driven Musculoskeletal Model
Muscle ContracUon Dynamics + Hill Model
a(t) F EMT (θ,t)
Joint angles: θ(t)
EMG
Model Parameters (EsUmated from CalibraUon)
PH
IMU LMT (θ,t)=f (θ,t)
LMT (θ,t)
NRMSE=7.3 ± 1.0 Normalized Root Mean Square Error
Successes Limita>ons ü Fast (i7 Dual Core Intel <0.2s for 30s exercise)
with only 1% NRMSE degradaUon -‐> possibility of Real Time Analysis
Ø Firmware currently implemented only on MATLAB
ü Synchronized IMU + EMG Ø EKF opUmized only for the upper limbs
Synchronized
Research approach and activities
• Human-Robot interaction
• Healthcare • And many
many more…
External Environment
Internal Environment
Autonomic Reflex
Robot
Sensing
Sensing Personality
Emotion Need
Motion
Motion
Expression Personality
Emotion Need
Emotion Mood
Need
Consciousness
Original Mental Model
Motion Reflex Sensing
Recognition Behavior Intelligence
External Environment
Internal Environment
Autonomic Reflex
Robot
Sensing
Sensing Personality
Emotion Need
Motion
Motion
Expression Personality
Emotion Need
Emotion Mood
Need
Consciousness
Modified Mental Model
Motion Reflex Sensing
Recognition Behavior Intelligence
Sensing
Human-Robot Interaction Example:
Stress management
Artist
Performance rule Modulation of the musical output
of the robot by an artist`s performance
Performance rule
Robot Performance feedback
Interaction System General Structure
Performance rule
Robot Performance feedback Artist
Human-Robot Interaction Example:
Performance Start Synchronization
Dance performance: very strong expression of instinctive motion based on musical performance.
Robot Performance feedback
Artist
Instinctive interaction
Performance rule
Dancer
Human-Robot Interaction Example:
Instinctive Interaction with a Dancer
Human-Robot Interaction Example:
Directing a musical robot
Music
Robot (WF4-‐RVI)
Conductor Data Preprocessing
Ictus DetecUon
Tempo CalculaUon
Mood DiscriminaUon
MIDI Controller
WB-‐4
Beat SegmentaUon
Normal, Legato, Marcato
Tempo
Traditional training model
Skill Trainer Trainee
Evaluator/Expert
Score
Current universal evaluation model
Advice
Methodology proposal
Instrument / human moUon measurements
MoUon analysis and skill classificaUon
QuanUtaUve feedback
Training plaqorms Subject
Expert doctor
to develop a general skill evalua>on system • separated from the training devices • objecUve and quanUtaUve based on moUon analysis • adapUve to various training devices, surgical applicaUons and
clinical pracUce
Medical applications – Laparoscopy (1/3)
Qualitative analysis
Novices r-ECU
Experts r-ECU
Novices, differently from the experts, use the right forearm only during the second part of the exercise
Medical applications – Laparoscopy (2/3)
Quantitative analysis
Expert Surgeons Muscle Activation
Muscles present different behavior in different exercises 0
20
40
60
80
100
r-‐ECU r-‐FCR r-‐BB r-‐TB r-‐TP l-‐TP l-‐ECU l-‐FCR l-‐BB l-‐TB
Force [N]
Dry Boxes In vivo Experiment
** p<0.01
**
**
**
**
**
**
Medical applications – Laparoscopy (3/3)
Successes Limita>ons ü General system able to provide both
qualitaUve and quanUtaUve results Ø EKF opUmized only for the upper limbs
ü ApplicaUon in a complete curriculum for laparoscopic training
Ø Current HW setup invasive and uncomfortable
Take home message
Take home message
• We need to work hard for developing better robots
• More important, we need to work harder for understanding ourselves J
Human-Robot Emotional and Musical
Interactions Massimiliano Zecca
Professor of Healthcare Technology [email protected]
Special lecture for the International Journal of Advanced Robotic Systems