ISR – Institute of Systems and Robotics University of Coimbra - Portugal
description
Transcript of ISR – Institute of Systems and Robotics University of Coimbra - Portugal
![Page 1: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/1.jpg)
University of Coimbra
ISR – Institute of Systems and RoboticsUniversity of Coimbra - Portugal
Institute of Systems and Robotics
http://paloma.isr.uc.pt
![Page 2: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/2.jpg)
University of Coimbra
Determining face orientationfor a robot able to
interpret facial expressions
Carlos Simplício, José Prado and Jorge Dias
Presented by José Prado
2010
03 - 10
Human-Robot Interaction
![Page 3: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/3.jpg)
University of Coimbra
Summary
Human-Robot Interaction
Introduction(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)
(Structure of a DBN classifying facial expressions)
![Page 4: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/4.jpg)
University of Coimbra
Summary
Human-Robot Interaction
IIntroductionntroduction(Interactive Mobile Robots)(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)
(Structure of a DBN classifying facial expressions)
![Page 5: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/5.jpg)
University of Coimbra
Introduction
We are developing a service/assistant robot, an Autonomous Mobile Agent (AMA).
This agent, will be used in the context of assisted ambiance.
The global project addresses the emergent tendencies to develop new devices for assistance and services.
![Page 6: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/6.jpg)
University of Coimbra
Introduction
Human beings express their emotional states through:• facial expressions• gestures• voice• etc.
We propose:• a technique to determine face orientation based in human face symmetry;
• a DBN to classify human facial expressions.
![Page 7: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/7.jpg)
University of Coimbra
Introduction
The AMA must observe and react according facial expressions of a person.
Facial expressions recognition becomes easier if done in frontal face images.
The robotic system will be used to follow the human being movements and keeps always a frontal face.
![Page 8: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/8.jpg)
University of Coimbra
Summary
Introduction(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)
(Structure of a DBN classifying facial expressions)
![Page 9: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/9.jpg)
University of Coimbra
AMA - architecture
![Page 10: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/10.jpg)
University of Coimbra
Summary
Introduction(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)
(Structure of a DBN classifying facial expressions)
![Page 11: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/11.jpg)
University of Coimbra
Robotic System Controller - RSC
Robotic Platform movements:– Longitudinal translations;– Transversal translations;– Rotations.
Rotations correspond to an arc of circle centered in the human being.
Objective is to follow the rotation done by the human being, getting always an image of a frontal face.
Robotic Head can move in synchronization.
1
2
![Page 12: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/12.jpg)
University of Coimbra
Summary
Introduction(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)
(Structure of a DBN classifying facial expressions)
![Page 13: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/13.jpg)
University of Coimbra
Face Pose Identification System - FPIS
In a perfect symmetric image,pixels positioned symmetricallyhave the same gray-level value: difference is zero.
We use this principle to verify if an image is symmetric: frontal face.
Example 1
Example 2
![Page 14: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/14.jpg)
University of Coimbra
Face Pose Identification System - FPIS
In a perfect symmetric image,pixels positioned symmetricallyhave the same gray-level value: difference is zero.
Problems:• By nature, human faces are not perfectly symmetric;• There are shadows.
But it works!!!
![Page 15: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/15.jpg)
University of Coimbra
Face Pose Identification System - FPIS
Define a vertical axis (always in the same position);
Calculate differences of gray-levels between symmetric (position) pixels. Build Normalized Gray-level Difference Histogram (NGDH).
In a frontal face, the vertical axis bisects the face and the information collected in the NGDH is strongly concentrated near the mean.
Else, the information is scattered along the NGDH.
NGDH withscattered information
NGDH withconcentrated information
![Page 16: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/16.jpg)
University of Coimbra
Face Pose Identification System - FPIS
Algorithm:Find and extract face region in the image;
Define a vertical axis (dividing the region in two parts with equal number of pixels);
Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;
Built NGDH's;
Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;
Synthesized image with great pseudomean has the frontal face!!
![Page 17: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/17.jpg)
University of Coimbra
Face Pose Identification System - FPIS
![Page 18: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/18.jpg)
University of Coimbra
Face Pose Identification System - FPIS
Algorithm:Find and extract face region in the image;
Define a vertical axis (dividing the region in two parts with equal number of pixels);
Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;
Built NGDH's;
Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;
Synthesized image with great pseudomean has the frontal face!!
![Page 19: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/19.jpg)
University of Coimbra
Face Pose Identification System - FPIS
![Page 20: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/20.jpg)
University of Coimbra
Face Pose Identification System - FPIS
Algorithm:Find and extract face region in the image;
Define a vertical axis (dividing the region in two parts with equal number of pixels);
Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;
Built NGDH's;
Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;
Synthesized image with great pseudomean has the frontal face!!
![Page 21: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/21.jpg)
University of Coimbra
Face Pose Identification System - FPIS
![Page 22: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/22.jpg)
University of Coimbra
Face Pose Identification System - FPIS
![Page 23: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/23.jpg)
University of Coimbra
Face Pose Identification System - FPIS
Algorithm:Find and extract face region in the image;
Define a vertical axis (dividing the region in two parts with equal number of pixels);
Synthesize face images - use vertical axis to perform a 3D transformation (rotation);Synthesized images are “hypotheses” to find the face out-of-plane rotation;
Built NGDH's;
Find the pseudomean – number of occurrences in a narrow region around the NGDH's mean;
Synthesized image with great pseudomean has the frontal face!!
![Page 24: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/24.jpg)
University of Coimbra
Face Pose Identification System - FPIS
![Page 25: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/25.jpg)
University of Coimbra
Face Pose Identification System - FPIS
OriginalAngle = 0º
Rotation
0º
Rotation
+30º
Result -30º
Result 0º
Result +30º
Rotation
-30º
![Page 26: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/26.jpg)
University of Coimbra
Face Pose Identification System - FPIS
OriginalAngle = -30º
Rotation
-30º
Rotation
0º
Rotation
+30º
Result -60º
Result -30º
Result 0º
![Page 27: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/27.jpg)
University of Coimbra
Face Pose Identification System - FPIS
OriginalAngle = +30º
Rotation
-30º
Rotation
0º
Rotation
+30º
Result 0º
Result +30º
Result +60º
![Page 28: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/28.jpg)
University of Coimbra
Summary
Introduction(Interactive Mobile Robots)
Autonomous Mobile Agent (AMA)Robotic System Controller (RSC)
Face Pose Identification System (FPIS)
Automatic Facial Expressions Recognition System (AFERS)Automatic Facial Expressions Recognition System (AFERS)
((Structure of a DBN classifying facial expressions)Structure of a DBN classifying facial expressions)
![Page 29: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/29.jpg)
University of Coimbra
Facial Expressions
We only consider five emotional states.
Each emotional state has a characteristic facial expression.
A facial expression is a set of Action Units (AUs).
Paulo José Carlos Carlos Alex
anger fear happy neutral sad
AUs are “distortions” of facial features.
Ex: lips smile.
![Page 30: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/30.jpg)
University of Coimbra
DBN's Structure
![Page 31: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/31.jpg)
University of Coimbra
DBN's Structure
Level 1
Emotional States considered are: anger fear happy sad neutral other
Node (variable) that probabilistically reflect the existence of an Emotional State.
otherneutral,sad,happy,fear,anger,StateEmotional
![Page 32: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/32.jpg)
University of Coimbra
DBN's Structure
Level 2
Expressions considered are: anger fear happy sad neutral
Nodes (variables) that probabilistically reflect the existence of a facial expression.
yesno,AngerE
yesno,NeutralE
...
![Page 33: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/33.jpg)
University of Coimbra
DBN's Structure
Level 311 AUs are considered in each facial expression.
![Page 34: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/34.jpg)
University of Coimbra
DBN's Structure
Level 4
Nodes (variables) that probabilistically reflect the strength of the evidences (positive or negative).
maximumhigh,medium,low,minimum,AU1EH
maximumhigh,medium,low,minimum,AU25EH
...
![Page 35: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/35.jpg)
University of Coimbra
DBN's Structure
Level 5
Here, information is propagated between time slices.
These nodes (variables) combine / fuse probabilistically, through inertia, information coming from the low level in present time slice with that from the previous instant.
maximumhigh,medium,low,minimum,tAU1SensorEH
maximumhigh,medium,low,minimum,tAU1SensorEH 1
...
...
![Page 36: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/36.jpg)
University of Coimbra
DBN's Structure
Level 6
Nodes (variables) collecting the evidences provided by the sensors.
maximumhigh,medium,low,minimum,AU1SensorEH
maximumhigh,medium,low,minimum,AU25SensorEH
...
![Page 37: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/37.jpg)
University of Coimbra
Conclusions
It was developed:
An architecture for an Autonomous Mobile Agent;
A Face Orientation Identification Technique;
A structure for a DBN.
The Face Pose Identification Technique has a good performance and is very fast.
Classification of facial expressions using positive and negative evidences is very promising.
![Page 38: ISR – Institute of Systems and Robotics University of Coimbra - Portugal](https://reader036.fdocuments.us/reader036/viewer/2022081520/568159fb550346895dc74884/html5/thumbnails/38.jpg)
University of Coimbra
END
Thanks for your attention!!!
Questions ?