FACE DETECTION USING SKIN COLOR AND EIGENFACE...
28
FACE DETECTION USING SKIN COLOR AND EIGENFACE TECHNIQUES FOR HUMAN-ROBOT INTERACTION WONG HWEE LING UNIVERSITI TEKNOLOGI MALAYSIA
Transcript of FACE DETECTION USING SKIN COLOR AND EIGENFACE...
FACE DETECTION USING SKIN COLOR AND EIGENFACE TECHNIQUES
FOR HUMAN-ROBOT INTERACTION
WONG HWEE LING
UNIVERSITI TEKNOLOGI MALAYSIA
FACE DETECTION USING SKIN COLOR AND EIGENFACE TECHNIQUES
FOR HUMAN-ROBOT INTERACTION
WONG HWEE LING
A thesis submitted in fulfilment of the
requirements for the award of the degree of
Master of Engineering (Electrical)
Faculty of Electrical Engineering
Universiti Teknologi Malaysia
MAY 2005
iii
Dedicated to my beloved family
iv
ACKNOWLEDGEMENT
First and foremost, I would like to express my deepest gratitude and
appreciation to my supervisor, Prof. Dr. Shamsudin H. M. Amin. His devotion to the
needs of his students, his encouragement and tolerance have make working with him
a priceless experience. This thesis is made possible through his constant support.
My sincere appreciation is extended to everyone from UTM’s Mechatronic
and Robotic Department for their advice and commentaries. A million thanks to
those few who have patiently taught me to operate the tools and equipments.
My earnest gratitude also goes to Assoc. Prof. Dr. Syed Abdul Rahman, Dr.
Edmund Koh, Ms. Eileen Liew and many others for sharing their knowledge freely. I
am also grateful to everyone who has willingly participated in my data collection
sessions and experiments.
Without exception, I would like to express my gratitude to Malaysian
Ministry of Science, Technology and the Innovation, which has partially supported
this project under the Intensification of Research in Priority Areas grants. I also
acknowledge Freescale Semiconductor Malaysia Sdn. Bhd. (formally known as
Motorola Malaysia Sdn. Bhd.) for providing postgraduate studentship support during
my time-span of study.
Lastly, warmest regards are sent to my family members and many friends for
their seamless caring and moral support. Above all, I thank God for making this
journey possible.
v
ABSTRACT
The main objective of the research undertaken here is to develop an
automated face detection system to be implemented on a mobile robot for human-
robot interaction (HRI) applications. HRI has gained increasing attention in recent
years due to escalating numbers of assimilation of robots into human lives. Face
detection is a vital aspect to sense human presence in a robotic environment. Despite
the fact that research on face detection has long been carried out, development and
implementation of face detection on mobile robot remains a great challenge due to
the complexity of the robot itself. A HRI model is presented to represent a human-
robot relationship in the shared robotic environment. The work focuses in building a
natural form one human to one robot HRI based on face detection. The main concern
of the work is to develop face detection using multiple features to increase robustness
of the system. The face detection comprises of a skin color detection based on
various chrominance information of images using multiple color spaces. Later, the
skin region is verified using crisp rules face verification. Different rules are applied
at different robot behavior that accommodates the HRI. Where applicable, eigenface
method is implemented for further verification. Later, the robot makes decision and
reacts based on the face detection result using heuristic search on the decision tree.
An adaptive weighting technique for the branches in the decision tree is presented to
solve problem for various robot behaviors and noise in images. Graphical user
interface (GUI) and text-to-voice ability are employed on the robot to support HRI.
Through experiments, it appears that the robot can successfully detect human face
and react meaningfully towards the face that has been detected.
vi
ABSTRAK
Tujuan utama penyelidikan ini dijalankan adalah untuk menghasilkan satu
sistem automasi pengesanan muka manusia yang ditempatkan pada sebuah robot
mudah alih untuk aplikasi interaksi antara manusia dengan robot. Interaksi antara
manusia dengan robot semakin mendapat perhatian kerana bilangan robot yang
diasimilasikan dalam kehidupan manusia meningkat. Pengesanan muka penting
untuk mengesan kehadiran manusia dalam persekitaran robotik. Walaupun
penyelidikan dalam bidang pengesanan muka telah lama dijalankan, pembangunan
and aplikasi pengesan muka pada robot mudah alih sukar disebabkan kompleksiti
pada robot tersebut. Satu model diberi sebagai satu gambaran untuk hubungan
manusia-robot dalam sebuah lokasi yang dikongsi bersama. Kerja ini fokus pada
pembangunan interaksi antara satu manusia dengan satu robot secara semulajadi
menggunakan pengesanan muka. Kerja ini menitikberatkan pembangunan pengesan
muka mengunakan pelbagai ciri untuk meningkatkan prestasi sistem tersebut.
Pengesan muka itu terdiri daripada pengesan kulit manusia melalui variasi maklumat
krominasi pada imej menggunakan pelbagai ruang warna. Kemudian, sektor kulit
tersebut akan dikenalpasti jika ia adalah muka dengan menggunakan peraturan tetap.
Peraturan adalah berbeza untuk tingkah laku robot yang berlainan bagi menampung
interaksi manusia dengan robot. Jika berkenaan, cara eigenface akan diaplikasikan
untuk proses pengenalpastian selanjutnya. Kemudian, robot membuat keputusan dan
memberi reaksi berdasarkan pencarian pintar pada pokok keputusan. Satu teknik
pemberat secara adaptif untuk cabang pada pokok keputusan diberi untuk
menyelesaikan masalah pelbagai tingkahlaku robot dan kebisingan dalam imej.
Antaramuka pengguna secara grafik dan kemampuan penukaran ayat kepada suara
diaplikasi pada robot untuk menyokong interaksi manusia dengan robot. Melalui
eksperimen-eksperimen yang dijalankan, robot tersebut dapat mengesan manusia
dengan berjaya dan memberi reaksi yang bermakna kepada muka yang dikesan.
vii
TABLE OF CONTENTS
CHAPTER TITLE PAGE
DECLARATION ii
DEDICATION iii
ACKNOWLEDGEMENT iv
ABSTRACT v
ABSTRAK vi
TABLE OF CONTENTS vii
LIST OF TABLES xii
LIST OF FIGURES xiii
LIST OF ABREVIATIONS xvi
LIST OF APPENDICES xx
1 INTRODUCTION 1
1.1 Introduction to Robotics 2
1.2 Research Background 3
1.3 Problem Background 5
1.3.1 Structural Relationship of Human, Robot
and the Environment 5
1.3.2 Natural Human-Robot Interaction (HRI) 6
1.3.3 Challenges in Robot Design 6
1.4 Research Objectives 7
1.5 Scope of Project 7
1.6 Outline of Thesis 8
viii
2 LITERATURE REVIEW 9
2.1 Face Detection 10
2.1.1 Challenges in Face Detection 11
2.1.2 Overview of Face Detection Methods 12
2.1.2.1 Face Features 13
2.1.2.2 Skin Color Detection 15
2.1.2.3 Face and Non-face Classifications 16
2.1.3 Comparison on Face Detection Methods 18
2.1.4 Application of Face Processing in Robotics 19
2.2 Human-Robot Interaction (HRI) 21
2.2.1 Human-Computer Interaction (HCI) 22
2.2.2 Overview of HRI Techniques 23
2.2.3 Guidelines and Issues in HRI 26
2.3 Intelligent Robots 28
2.3.1 Artificial Intelligence (AI) 29
2.3.2 Robot Vision 30
2.3.3 Guidelines and Issues in Mobile Robot Design 31
2.4 Discussions 33
2.5 Summary 34
3 SYSTEM DEVELOPMENT METHODOLOGY 35
3.1 Design Methodology 36
3.1.1 Application Initializations 37
3.1.1.1 Human Initialization 37
3.1.1.2 Mobile Robot Initialization 38
3.1.2 Human-robot Interaction (HRI) Description 38
3.1.3 Data Acquisition and Processing 40
3.1.4 Performance Analysis 41
3.1.4.1 Mobile Robot Performance 41
3.1.4.2 Face Detection Performance 43
ix
3.1.4.3 Human-robot Interaction (HRI)
Performance 44
3.2 Human-Robot Interaction (HRI) Model 45
3.2.1 Physical Level 46
3.2.1.1 Vision Unit 48
3.2.1.2 Motor Unit 48
3.2.1.3 Control and Processing Units 49
3.2.1.4 Interfacing and Power Units 49
3.2.2 Syntactic Level 50
3.2.3 Semantic Level 50
3.2.4 Intelligent Search for System Integration 52
3.3 Summary 54
4 FACE DETECTION DEVELOPMENT 55
4.1 Skin Color Detection 56
4.1.1 Color Space Conversions 58
4.1.1.1 YUV Color Space 58
4.1.1.2 Normalized RGB Color Space 60
4.1.1.3 CIE’s XYZ Color Space 61
4.1.1.4 Modified HSV Color Space 62
4.1.2 Skin Color Analysis 63
4.1.3 Image Processing for Skin Color Detection 65
4.1.3.1 Preprocessing 66
4.1.3.2 Skin Color Detection Based On
Multiple Color Components 69
4.1.3.3 Post-processing 74
4.1.4 Comparison for Skin Color Detection 74
4.1.4.1 Comparison Between One Color Space
and Two Color Spaces Skin Color
Detection 75
4.1.4.2 Comparison for Different Threshold
Values 79
x
4.2 Face Verification: Crisp Rule 80
4.3 Face Verification: Eigenface 84
4.3.1 Training Images and Test Images Segmentation 85
4.3.2 Principal Component Analysis 87
4.3.3 Distance to Face Space 89
4.4 Summary 90
5 FACE DETECTION IMPLEMENTATION AND
SYSTEM INTEGRATION 91
5.1 Heuristic Search on Decision Tree 92
5.1.1 Decision Tree for Face Detection 92
5.1.2 Adaptive Weighting for Decision Tree 94
5.2 Design of Behavior Modules 97
5.3 Input-output Description 99
5.3.1 Graphical User Interface (GUI) 99
5.3.2 Face Detection and Robot’s Reaction 102
5.4 Summary 107
6 EXPERIMENTAL RESULTS AND DISCUSSION 108
6.1 Experiments and Discussions for Offline Face
Detection 110
6.1.1 Experiments for Face Detection via Rule-based 110
6.1.1.1 Comparative Result Using Different
Color Spaces 112
6.1.1.2 Robustness of Face Detection Using
Multiple Color Spaces 117 6.1.2 Experiments for Face Detection via Eigenface 123
6.1.2.1 Comparison For Eigenface Face Detection 125
xi
6.1.2.2 Improvement of Rule Based Face
Detection Based on Eigenface 130
6.1.3 Discussions 132
6.2 Experiments and Discussions for Online Face Detection 133
6.2.1 Experimental Results for Rule-based Face
Detection 133
6.2.2 Experimental Results for Eigenface Face
Detection 141
6.2.3 Discussions 143
6.3 Onboard Face Detection 144
6.3.1 Behavior before sensing human face 144
6.3.2 Behavior after detecting one human face 145
6.3.3 Behavior during interaction with human 145
6.3.4 Behavior after human exited the program 146
6.3.5 Discussions 147
6.4 Summary 148
7 CONCLUSIONS 149
7.1 Research Summary 150
7.2 Research Contributions 152
7.2.1 Face Detection Implementation on Mobile Robot
based on HRI Model Using A Heuristic Search Technique on Decision Tree 152
7.2.2 Face Detection Development Using Multiple Features 153
7.3 Future Developments 154
7.4 Summary 155
REFERENCES 156
APPENDICES A-D 166
xii
LIST OF TABLES
TABLE NO. TITLE PAGE
2.1 Features for face detection 14
2.2 Classifications for HRI design 25
3.1 Overview on specification of HRI attributes 39
4.1 Histogram for original image and segmented images 69
4.2 Upper and lower threshold values for various chrominance
components 79
6.1 Performance for face detection during offline and online
experiments 109
xiii
LIST OF FIGURES
FIGURE NO. TITLE PAGE
2.1 General overview of studies for face detection based HRI 9
3.1 The sequence in the design stages 36
3.2 Block diagram for robot’s using microcontroller control 41
3.3 Block diagram for robot using computer control 42
3.4 Block diagram for vision based mobile robot using computer control 43
3.5 Human-robot interaction (HRI) model for face detection based HRI 46
3.6 Photograph of the robot prototype and its subsystems 47
3.7 Onboard face detection on mobile robot 50
3.8 Human-robot interaction in-progress 51
3.9 A simple tree-based graphical structure 53
4.1 Overview of the face detection development process 55
4.2 Standard deviation for skin pixels of individual persons 64
4.3 Standard deviation for combination skin pixels 65
4.4 Skin color detection using a pair of chrominance component 66
4.5 Converting two-dimensional pixel array into one-dimensional array 67
4.6 Blurring process using a 3x3 convolution mask 68
4.7 Images in separate color components 68
4.8 Histogram threshold 70
4.9 Threshold result of different chrominance images 71
4.10 AND operation on binary values of pixels 72
4.11 AND process to combine binary U image and binary V image 72
4.12 Median filtering of a U-V image 73
4.13 Detection and display of the initial SROI 74
4.14 Flow chart to detect skin color using U-V or r-b chrominance pair 76
xiv
4.15 Flow chart to detect skin color using various chrominance pair 77
4.16 Comparison of nearby-frontal face detection result using different
color spaces 78
4.17 Comparison of threshold between average values and proposed values 80
4.18 Width and height of initial SROI 81
4.19 SROI of face images with different head pose (U-V) 81
4.20 SROI of face images with different head pose (without neck) 82
4.21 False face detection of SROI images using Rule Set A 84
4.22 Masking process of a segmented face image 85
4.23 Histogram equalized training image 86
4.24 Training image collection for original SROI images that are face 88
5.1 Decision tree for face detection implementation 93
5.2 Initial weighting for different chrominance pairs 95
5.3 Stacking weights to determine the chrominance pair to be processed 96
5.4 Behavior flow for human-robot interaction 98
5.5 GUI for face detection-based HRI application 101
5.6 Password protection for variables initialization 101
5.7 Robot’s action in Mode I 103
5.8 Distance based on size of FROI 104
5.9 Active and neutral region in robot’s camera view 105
5.10 Reaction of robot in various active regions 105
5.11 Face center location in active region 106
5.12 Face center location in neutral region 107
6.1 Offline skin color detection based face detection platform 111
6.2 Examples of face detection 111
6.3 Accuracy of face detection using Rule Set A based on single
chrominance pair 112
6.4 Accuracy result of different person using U-V chrominance pair 113
6.5 Comparison of accuracy between different chrominance pairs
of Person 6 and Person 11 114
6.6 Comparison of result between multiple chrominance pairs-based
face detection with single chrominance pair-based face detection 115
6.7 Average computation time of each image frame for face detection
based on various chrominance pairs 115
xv
6.8 Processing time for face detection using U-r chrominance pair for
each image frame 116
6.9 Multiple color spaces face detection on human with different skin
tones 117
6.10 Face detection rate for face with different size and pose 118
6.11 Face detection of human with various face size and pose 118
6.12 Face detection for frontal near face image with various facial
expressions 120
6.13 Face detection for images captured with changes in ambient light 121
6.14 Face detection in images that consist of more than one people 122
6.15 Offline eigenface face detection platform 123
6.16 Color indication of face detection using eigenface 124
6.17 Accuracy for face detection based on Experiment 1 126
6.18 Accuracy for face detection based on Experiment 2 126
6.19 Accuracy for face detection based on Experiment 3 127
6.20 Accuracy for face detection based on Experiment 4 127
6.21 Computation time for face detection based on Experiment 2 128
6.22 Training images and eigenface for eigenface face detection 129
6.23 Eigenface face detection result 129
6.24 Comparison of face detection using Rule Set A with and without
eigenface face detection 131
6.25 Accuracy of online rule-based face detection on images with
single face 134
6.26 Face detection and distance verification for 20 test subjects 135
6.27 Accuracy of face detection and face center detection on images
with single face 136
6.28 Face detection and face center location for images with single face 137
6.29 Face detection in images with more than one human presence 138
6.30 Non-face detection in random images 139
6.31 Accuracy of online rule-based face detection 140
6.32 Accuracy for online face detection based on Rule Set D and
eigenface 141
6.33 Correct online face detection result in images 142
xvi
LIST OF ABBREVIATIONS
A/D - Analog-to-digital converter
Ah - Ampere per hour
AI - Artificial intelligence
API - Application program interface
Av i - are the eigenvectors of covariance matrix C
b - Blue component in normalized RGB color space
B - Blue component in RGB color space
BLDC - Brushless direct current
cm - Centimeter
col - total number of pixels in x-axis of two-dimensional array
C - Color vector with non-zero luminance
C - Covariance matrix,
C' - Radial projection of C in Maxwell
CCD - Charge-coupled device
CIE - Commission International de I’ Èclairage
DARPA - Defense Advanced Research Projects Agency
DC - Direct current
DIFS - Distance to face space
DIO - Digital input-output
DLL - Dynamic link library
DOF - Degree of freedom
e2 - Euclidean distance
ETA - Electronic travel aids
fps - Frames per second
F - face
FALSE - False face detection
FROI - Face region of interest
xvii
G - Green component in RGB color space
GHz - Giga hertz
GUI - Graphical user interface
h - height of initial SROI
hnew - New height of segmented SROI
hold - Old height of SROI
H - Hue component in Modified HSV color space
HCI - Human-computer interaction
HMI - Human-machine interaction
HRI - Human-robot interaction
HSV - Color space for specification in terms of hue, saturation and value
I - Intensity component in Modified HSV color space
I/O - Input-output
IQ - Intelligent Quotient
j - Location of pixel position in one-dimensional array
LAB - Color space with a perceptually uniform coordinates in 3-dimensional
color space
LCD - Liquid crystal display
min - Minimum value of pixels in one image
max - Maximum value of pixel in one image
MB - Mega byte
MFC - Microsoft Foundation Class
MISSED - Missed face detection
MRF - Markov random field
n - 1, 2, 3, …, number of pixels
n - 15 different skin images
n Face - Number of face detected by human
n False - Number of non-face images detected by computer
n Frames - Number of frames process in T
n Missed - Number of face not detected by computer
n True - Number of face correctly detected by computer
nm - Nano meter
N - Number of elements in the one dimensional image vector
NF - Not face
xviii
NS - Not skin color
NN - Neural network
NSF - National Science Foundation
M - 1, 2, 3, …, number of images
PCA - Principal component analysis
PCMCIA - Personal Computer Memory Card International Association
Qc - Quasi-chroma component in Modified HSV color space
r - Red component in normalized RGB color space
R - Red component in RGB color space
RAM - Random Access Memory
RGB - Red, green, blue color space
RIA - Robotics Industries Association
s - Standard deviation
SDK - Software development kit
SROI - Skin region of interest
SVM - Support vector machine
t - Time for robot to change position
t0 - Unique positive real number
th - Threshold value for Euclidean distance
T - Total time utilized to process a series of image frame
TRIPOD - Template for Real-time Image Processing Development
TRUE - True face detection
ui - Eigenface
U - U component in YUV color space
USB - Universal serial bus
v - Volt
vi - Eigenvectors of ATA
V - V component in YUV color space
w - Width of initial SROI
w - Weight
wold - Old width of SROI
x - X component in CIE’s XYZ color space
xyz - Color space with a perceptually uniform coordinates in chromaticity
plane
xix
x - Location in x-axis of two-dimensional array
X - Pixel value
- Mean of all pixel values
y - Luminance component in CIE’s xyz color space
y - Location in y-axis of two-dimensional array
Y - Luminance component in YUV color space
YCbCr - Color space for a digital color system
YIQ - Color space for encoding a composite video signal where Q
component has narrower bandwidth of spectrum than I component
YUV - Color space for encoding a composite video signal where UV
components occupy the same bandwidth of spectrum
z - Z component in CIE’s XYZ color space
ρ, α, β - Positive real values used in RGB to CIE’s XYZ color space
conversion
µi - Eigenvalue
Γ - Training set of face database
Γnew - Input face image
Ψ - Average image
Φ - Image difference
X
xx
LIST OF APPENDICES
APPENDIX TITLE PAGE
A Flow Chart of the GUI application program 166
B Operation Manual for Face Detection-Based Mobile Robot 170
C Electronic Circuits for Robot 181
D Video Clips for the Face Detection and Its Implementation on
Mobile Robot 185
CHAPTER I
CHAPTER 1
INTRODUCTION
Studies in the area of robotics have been growing swiftly for the past decade
and the ir developments are even faster at the turn of this millennium. Previously,
researchers within the robotics community have been emphasizing more on technical
challenges of achieving mobility, control and intelligence rather than social issues.
Lately, the development of robotics has motivated the feasibility of
integrating robots into human daily lives. Therefore, study on human-robot
interaction (HRI) is taken to more serious consideration in research within the
robotics community. As robots have to interact with humans, it is important to detect
humans in the first place. The research presented here focused on developing a
natural HRI system based on face detection that integrates a HRI model to clarify the
human-robot relation.
Section 1.1 presents introduction to robotics. Next, section 1.2 describes on
research background of HRI. The problem background of developing a HRI system
is mentioned in section 1.3. By identifying the problem, objectives of the study are
highlighted in section 1.4. Meanwhile, the scope of the project is explained in
section 1.6. Later, section 1.7 presents the outline of thesis. Last but not least, the
summary of Chapter 1 is portrayed in section 1.8.
2
1.1 Introduction to Robotics
Robots are moving machines created by men. People, when mentioned about
robots, will certainly thought of movies such as War of the Worlds, Star Wars,
Robocop, Terminator, Matrix and most recently I-Robot. The term “robot” originates
from a play staged in London, called "Rossum's Universal Robots" written by the
Czech writer, Karel Çapek, in 1921. Robot is derived from a Czech word robota,
which means a servitude or forced worker (Groover, 1986).
In general definition, a robot can be described as an automated machine,
performing functions in human manner, automation and mechanical slave (Webster,
1995). Robotics Industries Association (RIA)’s defined a robot as a reprogrammable,
multifunctional manipulator designed to move materials, parts, tools, or special
devices through variable programmed motions for the performance of a variety of
tasks (Groover, 1986). Arkin (1998) provided another working definition of robots,
which is inclusive of mobile robots. According to him, an intelligent robot is a
machine able to extract information from its environment and use knowledge about
its world to move safely in a meaningful and purposive manner.
Human and robots have a strong correlation ever since the emergence of the
very first idea of robotics. All robots fulfill one purpose, which is to serve mankind.
Isaac Asimov listed the basic principles of robot and human relationship and how the
robot should behave in Three Laws of Robotics (Groover, 1986). The laws stand as
an informal universal rule on how robot should conduct itself in a human world.
Throughout the years, robots are changing in shape, size and mobility method.
Along with their physical evolution, issues of robot intelligence come into the
limelight in order for a robot to behave and act smartly in a real world. Nevertheless,
with strong development in artificial intelligence (AI), robots are claimed to have
ability to think, learn and evolve by their own (Pfeifer, 1999). According to Hans
Moravec (1991), robots will have human perceptual, motor abilities and their own
superior reasoning powers after year 2050. Combining intelligence and
computational capabilities, will robots may become superior to human one day?
3
Until today, humans have not observed any robot that causes intentional
destruction to mankind. In fact, robots have aided human in many ways. Robots
provide support, companionship, entertainment, and etc. Eventually, study in
robotics stirs towards human-robot interaction (HRI), as more robots assimilate into
human daily life.
1.2 Research Background
Today, HRI has received much attention in robotics. According to Schlotz
(2003), HRI started with tele-operation for factory platforms. HRI is a cross
disciplinary area, that involves psychology, sociology, cognitive science,
communication and robotics. HRI refers to how human and robot interact in a given
situation. HRI has been a fraction of robotics research but it is still in its infancy as
researchers are more focused on achieving better control and mobility in robots
(Murphy and Rogers, 2001).
Service robots are sample of HRI and they are used for specialized task.
Service robot can be listed as exploratory robots, maintenance robots, military robots
and surgical robots. Those robots perform dangerous tasks or they can venture into
unsafe environments. Another type of service robots is more human oriented. They
are the office robots, home robots, hospital robots and robot tour-guides. The robots
tend to interact more with human and help people in a more personalized and
friendly way. The service robots hold responsibility to perform some serious task at a
specified time length. Thus, they are sometimes referred as the occupational robots.
Commercialization of mobile robots for general users can be found lately, as
the cost of making and marketing robots become more economical. Personal robots
are robots that can be purchased for individual usage. One of the personal robots is
the Electronic Travel Aids (ETA) for the visually impaired to navigate and avoid
obstacles (Borenstein and Ulrich, 1997). Other types of personal robots, which are
growing in quantity, are the robot toys. Aside providing service, the robots are able
4
to aid and offer entertainment to human. In other words, robots are able to help
human physically and emotionally.
Generally, the modalities for HRI are speech, vision and wireless techniques.
Wireless based HRI requires a communication tool in between the robot and human.
If a robot is in direct contact with a human, it must have some natural means of
communication (Sekmen, 2000). Therefore, speech and vision-based HRI are more
widely used for natural HRI development as no equipments or tools are needed in
between human and robot during the interaction process.
Speech processing is essential for robot to understand the verbal commands
given by a human. However, the robot must be capable to acknowledge the human
presence and location in order to interact with the person at the first place. Thus,
robot’s vision comes into limelight in HRI research.
Basically, a robot needs to ‘see’ a human in the robotic environment.
Through machine vision, knowledge of the environment is acquired and processed
by a robot and thus, providing a certain level of intelligence to the robot for HRI.
Vision-based techniques used in HRI are gesture recognitions using body gesture and
posture. Over the past few years, face processing has also emerged as an important
approach for HRI. Face processing methods includes face detection, face localization,
face tracking, face recognition and facial expression recognition.
In order to proceed to the mentioned face processing, it is crucial to have
good face detection ahead of it. Human face detection is taken as a natural
representation and visualization of a human existence in the robot’s environment.
According to Marsic et al. (2000), the face to be “seen” by the computer may be
complex but it is desirable to unburden a human to the extend possible. Through face
detection, a human does not have to speak or manipulate the robot physically in order
for robot to detect human presence. Due to the fact, face detection is an essential
ability to have in HRI robots.
5
1.3 Problem Background
The study begins by reviewing the impact of robots to human life. It is
without doubt that greater understanding of the vital link between human and robots
is essential when planning to integrate robots into human-centric robotic applications.
The human-centric robotic applications are basically the service robots and
personal robots, as those robots have to serve humans. Service robots and personal
robots applications are aimed for a novice user. In this context, a novice refers to a
potential human user of that robotic application, who has little or no experience of
handling robots. For this reason, HRI is also closely related to human-centric design.
Based on literature review, the study highlights some issues in HRI
development. They are generalized as the following:
1. Structural relationship of human, robot and the environment
2. Natural human-robot interaction
3. Challenges in robot design
1.3.1 Structural Relationship of Human, Robot and the Environment
Studies on HRI are important as robots become more human-centric.
Previously, robot researches basically emphasize on achieving intelligent control and
mobility. Besides the robot and its operating environment, human factor is integrated
into the robotic system. Human factor refers to human existence and human
intervention during HRI operation. Thus, information flow increases. A structural
relationship is vital to manage information sharing between human and robot for
decision making of both parties in the shared environment. Based on the literature
review, established organized structure for describing human-robot relation and
interaction could not be clarified.
6
1.3.2 Natural Human-Robot Interaction (HRI)
Natural relationship of human and robot in HRI is essential to create a level
of comfort during the process. Secondly, a robot that can interact naturally will tend
to attract human user. An example of a natural interaction is when a person talks to
another person face to face. In this case, the human and robot “talks” to each other
without any help from external equipment, which is neither part of body of the robot
nor the human. In human point of view, the interaction should be as natural as
possible to provide attraction and simplicity. As a whole, natural interaction can
provide a positive psychological factor for the interaction process.
1.3.3 Challenges in Robot Design
Reaction of the robot during HRI is affected by the challenges faced in robot
design. Similar to the challenges in general robotics, the main challenges in building
real robots are (Tan, 2002):
Robustness: Robot that operates in real world is succumbed to the
uncertainties in the environment, as prior knowledge of the environment may be
incomplete or non-existent. Knowledge gained from sensors may be incomplete,
inaccurate or irrelevant. The robot itself has limitation due to imperfection of its
sensors and actuators. Environment that is dynamic in nature also leads to
unpredictable changes. However, the robot must be able to react intelligently even
though it is governed by various limitations. Hence, robustness is essential to handle
the uncertainties.
Real-time responsiveness: A real-time system must satisfy explicit bounded
respond time constraint to avoid failure. A robot has to think and react within a
suitable period of time. In order to interact naturally, robot must be able to respond to
human in a “comfortable” time frame. However, comfortable time zone for each
person is dissimilar as every human expectation is inconsistent. However, the main
point is to avoid the robot to be in a state where its reaction is no longer meaningful.
7
1.4 Research Objectives
In summary, the main objective of the study is to develop an automated face
detection on a mobile robot for potential HRI implementation. The sub-objectives in
order to accomplish the main objective of the study are:
1. To propose a HRI model to manage information flow;
2. To build a mobile robot that can support a basic HRI based on face detection;
3. To develop a face detection application program that can operate in real-time;
4. To integrate the face detection on the mobile robot and test the
responsiveness of the system in the robot’s environment.
1.5 Scope of Project
The main purpose of the research is that it emphasizes on developing
automated face detection ability on mobile robot in an organized method for possible
HRI application by incorporating the proposed HRI model. The robot does not
utilize any other sensors to sense human presence. Hence, the robot mainly relies on
the face detection for motion and reaction.
The research is divided into three main modules. The initial work is the
construction of a HRI model. The second part involves development of a face
detection algorithm that is robust enough to function within the robot environment in
real-time. Improvement of face detection using skin color and eigenface techniques
is performed for the HRI application. The third part is testing of the developed
system in real world environment. A heuristic search technique is employed for the
robot to handle information, make decision and action based on the face detection
result. The responsiveness of the robot includes visualization of the face detection
result and robot’s written command on a graphical user interface, robot’s voice
command and its movement in the robotic environment.
8
1.6 Outline of Thesis
The thesis is organized in seven main chapters. Chapter 2 gives an overview
of the background studies on areas that are useful to build a face detection based
mobile robots for HRI. Next, detail of the proposed methodology is discussed in
Chapter 3. A HRI model is proposed to give a clear view of human robot relation.
Meanwhile, the face detection development methodology will be explained in details
in Chapter 4. The face detection comprises of initial face detection via skin color
detection and face verification via rule based and eigenface. Chapter 5 covers on
system integration of the face detection into a real mobile robot and the robot
reaction based on the information attained during HRI. Later, experimental results,
comparisons and discussions are presented in Chapter 6. Finally, Chapter 7 contains
the conclusions of study, which includes the contributions and future developments
of the work presented.
