Development and Implementation of Behaviours for a Socially Assistive Robot for the Elderly

By

Zhonghe (Jacob) Li

A thesis submitted in conformity with the requirements

for the degree of Masters of Applied Science

Mechanical and Industrial Engineering University of Toronto

© Copyright by Zhonghe Li 2015

Development and Implementation of Behaviours for a Socially

Assistive Robot for the Elderly

Zhonghe (Jacob) Li

Masters of Science

Mechanical and Industrial Engineering University of Toronto

2015

Abstract

Bingo is a cognitively stimulating recreational activity popular in long-term care facilities and

retirement homes. This thesis focuses on the development of the human-like socially assistive

robot Tangy for the autonomous facilitation of Bingo. A set of assistive social behaviours are

developed for multi-user and single user interactions the robot may encounter while facilitating

the Bingo game. In addition, development of actuation modules for establishing eye contact,

controlling the arms, and interacting with players using audio communication is discussed. A set

of experiments verified the system performance of these behaviours and actuation capabilities

and the efficacy of the social robot behaviours with elderly residents in long-term care. The

results demonstrate that the robot determines and executes the appropriate behaviours during the

facilitation of a Bingo game, and that elderly residents enjoyed interacting with Tangy and

wished to participate in Bingo games facilitated by the robot in the future.

ii

Acknowledgements

I would like to thank my supervisor, Professor Goldie Nejat, for her continued support, input, and

guidance throughout my research. I would also like to thank my M.A.Sc. thesis committee for their

time and feedback. I would also like to thank Geoffrey Louie, Chris Mohamed, Frank Despond,

Tiago Vaquero and Vincent Lee for their work on the collaborative Tangy project. Thank you to the

Dr. Robot Inc. developers for their generous technical support with the robot platform. I would like

to thank my friends and family, and my lab-mates for their advice and comradeship.

iii

Table of Contents

Acknowledgements ........................................................................................................................ iii

Table of Contents ........................................................................................................................... iv

List of Tables ............................................................................................................................... viii

List of Figures ................................................................................................................................ ix

Chapter 1 Introduction ..............................................................................................................1

1.1 Motivation ............................................................................................................................1

1.2 Tangy the Socially Assistive Robot .....................................................................................2

1.3 Problem Definition...............................................................................................................2

1.4 Proposed Methodology ........................................................................................................3

1.4.1 Literature Review.....................................................................................................3

1.4.2 Gathering Impressions/Design Considerations ........................................................3

1.4.3 Robot Behaviours.....................................................................................................3

1.4.4 Actuation Modules ...................................................................................................4

1.4.5 System Performance Review ...................................................................................4

1.4.6 HRI User Studies .....................................................................................................4

1.4.7 Conclusions ..............................................................................................................4

Chapter 2 Literature Review.....................................................................................................5

2.1 Cognitive Training Activities ..............................................................................................5

2.1.1 Specific Cognitive Training Programs .....................................................................5

2.1.2 General Cognitive Training Programs ....................................................................6

2.1.3 Bingo as a Cognitive Training Intervention .............................................................7

2.2 Socially Assistive Robots for Older Adults .........................................................................7

2.2.1 Single User Scenarios ..............................................................................................7

iv

2.2.2 Multi-User Scenarios ..............................................................................................8

2.3 Social Human-Robot Interaction .........................................................................................9

2.3.1 Eye Contact ............................................................................................................10

2.3.2 Physical Gestures ..................................................................................................11

2.4 Chapter Summary ..............................................................................................................13

Chapter 3 Gathering Impressions and Design Considerations ...............................................15

3.1 Focus Group Study ............................................................................................................15

3.1.1 Single User Scenarios ............................................................................................15

3.1.2 Participant Demographics .....................................................................................17

3.1.3 Thematic Analysis .................................................................................................18

3.2 Thematic Sets .....................................................................................................................18

3.2.1 Bingo .....................................................................................................................18

3.2.2 Telepresence ..........................................................................................................19

3.2.3 Tangy’s Appearance .............................................................................................19

3.2.4 Acceptance of the Robot .......................................................................................20

3.2.5 Ease of Use ............................................................................................................21

3.3 Suggested Features/Activities ............................................................................................22

3.3.1 Reminders/Prompting ............................................................................................22

3.3.2 Simple Interactions with Cognitively Impaired Residents ...................................22

3.3.3 Other Recreational Activities .................................................................................23

3.3.4 Physical Tasks and Aiding Residents’ Autonomy ................................................23

3.3.5 Music Therapy .......................................................................................................23

3.3.6 Multilingual Support .............................................................................................23

3.4 Chapter Summary ..............................................................................................................24

v

Chapter 4 Robot Behaviours...................................................................................................25

4.1 The Bingo Scenario............................................................................................................25

4.2 Behaviour Determination ...................................................................................................26

4.2.1 Sensory Capabilities ..............................................................................................27

4.3 Robot States During the Bingo Game ................................................................................26

4.3.1 Pre-Game State .....................................................................................................28

4.3.2 Multi-User State .....................................................................................................29

4.3.3 Transition State .....................................................................................................30

4.3.4 Single User State ...................................................................................................31

4.3.5 Post-Game State ....................................................................................................33

4.4 Behaviour Determination ...................................................................................................33

4.5 Chapter Summary ..............................................................................................................38

Chapter 5 Robot Actuation Modules ......................................................................................39

5.1 Arm Control Module..........................................................................................................39

5.1.1 Motion Planning for the Arms ...............................................................................39

5.1.2 Optimization Parameters ........................................................................................41

5.2 Eye Contact Module ..........................................................................................................43

5.3 Audio Interactions ..............................................................................................................43

5.4 Chapter Summary ..............................................................................................................45

Chapter 6 System Performance Review .................................................................................47

6.1 Finite State Machine Performance Review........................................................................47

6.1.1 Methods .................................................................................................................47

6.1.2 Results ....................................................................................................................48

6.2 Arm Control Accuracy Experiment ...................................................................................48

6.2.1 Methods .................................................................................................................48

vi

6.2.2 Results and Discussion ..........................................................................................52

6.3 Arm Control Accuracy Experiment ...................................................................................55

6.3.1 Methods .................................................................................................................55

6.3.2 Results and Discussion ..........................................................................................56

6.4 Chapter Summary ..............................................................................................................59

Chapter 7 Human Robot Interaction User Studies .................................................................60

7.1 Participants .........................................................................................................................60

7.2 Methods..............................................................................................................................61

7.3 System Performance Results ..............................................................................................62

7.4 Human-Robot Interaction Results......................................................................................63

7.4.1 Participant Questionnaire Results .........................................................................63

7.5 Discussion ..........................................................................................................................65

7.6 Chapter Summary ..............................................................................................................66

Chapter 8 Conclusion .............................................................................................................67

8.1 Summary of Contributions .................................................................................................67

8.1.1 Gathering End-User Feedback ..............................................................................67

8.1.2 Assistive Robotic Behaviours and Actuation Capabilities ...................................67

8.1.3 Experimental Results ............................................................................................68

8.2 Discussion of Future Work ................................................................................................68

vii

List of Tables

Table 1: Focus Group Questions....................................................................................................16

Table 2: Participant Demographics ................................................................................................17

Table 3: Computer and Robot Experience of Focus Group Participants .......................................17

Table 4: Pre-Game Behaviours ......................................................................................................28

Table 5: Multi-User State Behaviours ...........................................................................................30

Table 6: Transition Behaviours ......................................................................................................31

Table 7: Single user Behaviours ....................................................................................................32

Table 8: Post-Game Behaviour ......................................................................................................33

Table 9: Behaviour-Determining States.........................................................................................34

Table 10: Manufacturer Servo Specifications for Tangy's Arms .................................................43

Table 11: Finite State Machine Performance .................................................................................48

Table 12: Robot Right Arm Poses .................................................................................................51

Table 13: Accuracy Performance Review for Tangy's Right Arm ................................................53

Table 14: Accuracy Performance Review for Tangy's Left Arm ..................................................54

Table 15: Mean Errors in Joint Accuracy Performance over Three Trials ....................................55

Table 16: Duration and Energy Consumption Differences between Sets of Twenty-Five

Executions of Optimized and Un-Optimized Gestures ..................................................................58

Table 17: Number of Participants with Experience with Computers or Robots............................60

Table 18: System Performance Results .........................................................................................63

Table 19: Post-Bingo Session Questionnaire Results ....................................................................64

viii

List of Figures

Figure 1: Tangy the Socially Assistive Robot ...................................................................... 2

Figure 2: Tangy's System Architecture ..........................................................................................26

Figure 3: Off-board and On-board Sensors ...................................................................................27

Figure 4: Tangy’s FSM ..................................................................................................................36

Figure 5: Tangy Behaviours during Bingo Game ..........................................................................37

Figure 6: CAD Model of Tangy and Kinematic Model of Tangy’s Arm ......................................40

Figure 7: Face Detection Rotation Invariance ...............................................................................44

Figure 8: Face Tracking .................................................................................................................45

Figure 9: Servo Number Scheme (Right Arm) ..............................................................................49

Figure 10: Texas Instruments CC2650 SensorTag ........................................................................50

ix

1

Chapter 1 Introduction

1.1 Motivation:

As people age, they experience greater risks for cognitive impairments such as declines in

memory, abstract reasoning, attention skills and verbal skills [1]. These age-related cognitive

impairments among the elderly diminish their capability to independently perform activities of

daily living, such as eating, dressing and toileting [2]. Older adults with cognitive impairments

who are not able to live independently frequently turn to long-term care facilities where they can

receive support with these activities of daily living, medical services, as well as scheduled leisure

activities [3]. Long-term care facilities also provide the elderly with opportunities for social

engagement, which have been shown to improve the quality of life of long-term care residents

[4].

As the population is rapidly aging, the demand for long-term care facilities is growing larger and

larger [5]. Currently, the population of older adults above the age of 60 is 901 million, and is

projected to reach 3.2 billion by 2050 [6]. Conversely, the long-term care workforce, which

already traditionally sees higher rates of turnover, is facing lower numbers of healthcare staff

each year [5]. The diminishing long-term care workforce has led to gaps in service for elderly

residents, such as in the provision of recreational activities in these facilities [5]. The increasing

pressure on the long-term care health system has led researchers to explore assistive technologies

in order to provide aids for healthcare staff to lessen their workloads.

Cognitive training interventions are a form of therapy-based activities which are designed to

maintain or reduce the rate of age-related cognitive decline [7]. Various types of cognitive

training exist, with both specific and non-specific cognitive training activities [7]. Specific

cognitive training interventions target particular functions in the brain such as memory,

reasoning, or speed of processing [7]. Non-specific cognitive interventions do not target any

particular functions, but are designed to give general cognitive stimulation through recreational

activities [7].

1

2

1.2 Tangy the Socially Assistive Robot

One of the main objectives of the Autonomous Systems and Biomechatronics Laboratory

(ASBLab) is to develop and integrate robotic technologies in order to aid in cognitive

interventions for older adults. A key research area in the ASBLab focuses on developing the

socially assistive robot, Tangy, which can aid in facilitating general cognitive interventions in the

form of recreational activities. Tangy is a human-like socially assistive robot which is being

developed in order to autonomously facilitate the cognitively stimulating multi-user recreational

activity Bingo for retirement home and long-term care settings (Figure 1).

Figure 1: Tangy the Socially Assistive Robot

Currently, Tangy is being designed to directly incorporate common forms of communication

used in natural social interactions between people. The advantage of a socially assistive robot is

to be able to provide an intuitive and effective set of assistive behaviours upon employment in a

long-term care/retirement home setting due to its ability to communicate within existing social

structures. Moreover, the robot is capable of lessening the workloads of the healthcare staff,

while providing beneficial social and cognitive stimulation to elderly residents.

1.3 Problem Definition

The research behind this thesis focuses on the implementation of a socially assistive robot for

facilitating a multi-user cognitive intervention for older adults. In particular, Tangy will facilitate

a Bingo game by performing a set of structured behaviours for a group of elderly users. These

behaviours involve sensing the environment, determining a set of assistive actions based on the

2

3

state of the environment, robot and game, and performing these actions using a set of physical

and auditory actuation techniques. The objective of this thesis is to develop the decision-making

capabilities needed in order to generate the interactive robot behaviours for Tangy, as well as the

various software modules to determine and then physically display the social behaviours on the

robot. These modules discussed in detail in this thesis will include the robot deliberation, arm

motion planning, eye contact, and audio interaction systems.

1.4 Proposed Methodology

The overall methodology for the design of Tangy’s behaviours and actuation interfaces

comprises of the following components with corresponding reference to the thesis chapters:

1.4.1 Literature Review

In Chapter 2, literature review for the following areas, which are critical to the development of

socially assistive robots in long-term care, is presented: (i) cognitive interventions for older

adults; (ii) socially assistive robots used in healthcare settings for older adults; and (iii) social

behaviours implemented on robotic systems.

1.4.2 Gathering Impressions/Design Considerations

In Chapter 3, the results from focus group studies which gathered the impressions and

suggestions that residents, their family members and healthcare staff at a long-term care facility

and a retirement home had about socially assistive robots, including Tangy, are presented to

inform the design of the robot’s functionality and behaviours. The chapter discusses the

methodology of the focus groups, and then the various thematic sets which were drawn from the

discussions of the participants.

1.4.3 Robot Behaviours

In Chapter 4, a brief description of the Bingo multi-user scenario is first presented. Then, each

behaviour that the robot performs in the scenario is detailed. Lastly, the chapter presents the

development of the behaviour determination module for the robot, which focuses on choosing

the robot’s appropriate behaviours based on the current robot, game, and world states.

3

4

1.4.4 Actuation Modules

In Chapter 5, the design of the software interfaces used in Tangy’s physical implementation of its

behaviours is presented. These modular interfaces are described within the context of Tangy’s

overall layered system architecture. The functionalities of each layer as well as the individual

modular structure of each layer are explained. The inputs and outputs of each module are

outlined in order to present its function and role in the architecture. In particular, the section

describes the development of the arm control, eye contact, and audio interaction modules. With

respect to the arm control, both the motion planning and optimization techniques for choosing

the most efficient motion plan are discussed.

1.4.5 System Performance Review

In Chapter 6, experiments conducted in our lab to verify the success rates of the robot’s

behaviours during Bingo games and the optimization technique for choosing motion plans for the

robot’s gestures are presented. The optimization technique has been tested to ensure that the

optimized motion plans have lower time and energy costs when compared to a control set of

motion plans.

1.4.6 User Studies

In Chapter 7, detailed experiments and user studies at a long-term care facility and a retirement

home are presented to evaluate the implementation of the robot behaviours and the control

architecture for the intended population. User studies examined the use of the robot behaviours

in social interactions with elderly residents. Discussions to illustrate the effectiveness of the

proposed designs are also presented.

1.4.7 Conclusion

Lastly, Chapter 7 presents concluding remarks on the development of Tangy’s behaviours and

architecture, highlighting the main contributions of the thesis and future work.

4

5

Chapter 2 Literature Review

This section provides an overview of work related to socially assistive robots for older adults. In

Section 2.1, various types of cognitive training programs for the elderly are discussed,

concluding with a discussion about the recreational activity Bingo. In Section 2.2, literature on

socially assistive robots for single user and multi-user scenarios with the elderly is discussed.

Lastly, in Section 2.3, literature on social human-robot interaction is discussed.

2.1 Cognitive Training Activities

Cognitive training activities are therapy-based activities which aim to stimulate cognitive

functions in order to delay or halt age-related cognitive decline [1]. Cognitive training includes

both specific cognitive interventions and general cognitive training activities [7],[8],. Specific

cognitive training programs aim to stimulate particular functions such as memory, reasoning, or

speed of processing, while general cognitive stimulation activities involve general activities

which stimulate individuals through active cognitive and social engagement [7],[9].

2.1.1 Specific Cognitive Training Programs

Several specific cognitive training programs have been shown to reduce rates of cognitive

decline and improve functioning among those already suffering from cognitive impairments [10].

For example, the IMPACT (Improvement in Memory with Plasticity-based Adaptive Cognitive

Training) program was aimed to improve memory and attention in participants using exercises

self-administered on a computer [11]. The IMPACT training program was administered to older

adults above the age of 65, and included exercises to test their processing speed of speech-related

audio information. Results from the program indicated that there was a significant improvement

in auditory-based cognition when comparing participants who were administered the training

program and participants who participated in a control program.

Another specific cognitive training program was the ACTIVE (Advanced Cognitive Training for

Independent and Vital Elderly) program, which employed certified personnel to teach strategies

and administer exercises to improve performance in memory, reasoning and speed of processing

[8]. More specifically, the exercises included training verbal episodic memory, the ability to

5

6

solve problems which follow a serial pattern, and the ability to visually search and identify

information in a divided-information format. For training verbal episodic memory, participants

were taught mnemonic strategies and then given practice exercises related to everyday tasks. For

reasoning training, participants were given number or letter series and asked to identify the

patterns in the series. Finally, for speed-of-processing training, participants were asked to

perform speed tasks on a computer with continually increasing difficulties for the tasks. The

ACTIVE program was shown to improve the targeted cognitive abilities by magnitudes

equivalent to the declines in those abilities expected in an elderly person without dementia over 7

and 14 year periods (depending on the ability).

The success of these specific cognitive training programs often depends on adherence to self-

administered training and professional supervision. This reliance limits their availability for

institutionalized older adults [12].

2.1.2 General Cognitive Training Programs

General cognitive training programs which have been demonstrated to improve cognitive

functioning in older adults include Reality Orientation Training (ROT) [13], rehabilitative training

based on procedural memory [14], and/or recreational activities and art therapies [7],[16]. ROT

involves repetitively presenting the participant with orientation information (such as location or

time) throughout the day or in group meetings [13]. ROT has been demonstrated to significantly

improve Mini-Mental State (MMS) scores and other measures of cognition such as word list

memory for older adults [13],[14].

Rehabilitative training based on practicing activities of daily living has been used to stimulate

procedural memory [15]. This training has also been shown to provide improvement in the

performance of those activities by older adults and promote cognitive functioning [15]. After an

occupational therapist conducted procedural memory training with a group of participants for a

three week period by prompting, instructing and modelling thirteen activities of daily living, the

participants significantly increased their speed for performing those activities.

Participation in recreational activities has also shown benefits in cognitive functioning in the

elderly [7],[17]. Moreover, recreational activities provide secondary benefits such as greater

happiness, better physical functional abilities, and a reduction in mortality for older adults [18].

6

7

When elderly participants with mild Alzheimer’s disease participated in conversation, singing,

party games and dancing over six weeks as part of a cognitive training program, they experienced

improvements in behavioral and everyday functioning [7]. Another example of recreational

activities being used for cognitive training was presented in [17], where the AKTIVA (Aktive

Kognition Stimulation – Vorbeugung im Alter) program was used to educate and promote the

benefits of participating in cognitively stimulating recreational activities for older adults in the

community. Participants in the program were encouraged to read, play games, and play musical

instruments [17]. Results from the study showed the program enhanced speed of processing,

improved self-reported subjective memory decline, and increased voluntary participation in

cognitively stimulating activities.

2.1.3 Bingo as a Cognitive Training Intervention

A recreational cognitively stimulating activity that is popular among older adults is Bingo.

Bingo’s simplicity allows it to be accessible to a large range of player skills, and can be

increasingly challenging through the addition of cards a user simultaneously plays [19]. Bingo

has been shown to have therapeutic benefits for cognitive functions such as memory, recall, and

recognition [4]. Moreover, the activity provides a structured social setting which encourages

engagement among players. The social interactions players experience in a Bingo game are

important, as improving an elderly individual’s social network has a variety of benefits including

lowered mortality rates, reducing isolation and depression, and delaying age-related cognitive

decline [20],[21].

2.2 Socially Assistive Robots for Older Adults

Socially assistive robots have been designed as potential aids for facilitating cognitive training

programs for the elderly [22]-[27]. A variety of socially assistive robots have been designed to

engage in recreational activities with the elderly in both single user and multi-user scenarios. The

robots range from semi-autonomous facilitation of an activity with some assistance from a

human operator to fully autonomous.

2.2.1 Single User Scenarios

7

8

Socially assistive robots have been designed to engage in cognitively stimulating recreational

activities with the elderly such as memory card games [22], music games [23], and read-alongs

and arithmetic quizzes [24]. In [22], the human-like robot Brian 2.1 autonomously engaged

elderly individuals in a memory card game in which users memorize the locations of pairs of

unique cards which are briefly shown before being flipped over [22]. The study found that the

elderly users enjoyed interacting with Brian 2.1, thought that it was sociable, and were both

engaged in the interaction and compliant during the robot’s prompts. In [23], the small child-like

robot Bandit played a music-guessing game, in which excerpts of songs were played and

participants pressed the button corresponding to the song name, with cognitively impaired

elderly residents of a long-term care facility [23]. Through the activity, the study found that

Bandit was able to maintain and improve the impaired residents’ cognitive attention. In [24], the

small animated YORISOI Ifbot engaged older adults in conversational read-alongs of Japanese

folk tales and simple arithmetic calculations by speaking to the adults, asking questions and

instructing users to input their answers. This study found that all participants actively engaged in

the activities, and felt increasing affection for the robots.

2.2.2 Multi-User Scenarios

While socially assistive robots have been effective in engaging and instructing elderly residents

during single user activities as previously mentioned, those scenarios limit many of the social

benefits that can be achieved through multiplayer recreational activities. Multiplayer activities

instead can provide social engagement among players, which can reduce symptoms of

depression [20], decrease physical impairment [21], lessen the negative impact of widowhood on

health [21], and increase functional independence [21]. As such, socially assistive robots have

also been developed to facilitate or participate in recreational activities with multiple users.

Robots that are designed for multi-user scenarios with the elderly include: Ifbot, which facilitated

educational games [25]; Aibo, which played a physical catch game [26]; and Matilda, which

facilitated Bingo and Hoy games in [27].

In [25], a group of elderly long-term care residents in Japan participated in a study where

participants engaged in school-style recreational and academic activities with the Ifbot robot.

The activities were structured similar to primary school classes, with language studies, sing-

alongs, tongue twisters and arithmetic exercises. The robot would facilitate these various

8

9

activities by presenting exercises to the elderly users, receiving their solutions to the exercises,

and responding with an assistive behaviour based on their inputs. The interactions between the

robot and the participants were mediated by a human operator, who had to repeat the robot’s

prompts and manually input their responses into the robot. The results of the study indicated that

the robot and the activities were received positively, and the elderly participants actively engaged

in the exercises.

A different type of activity was investigated in [26], where an AIBO animal-like robot played a

physical recreational game with elderly nursing home residents in Japan with moderate

dementia. The game was a three-fold ball-catching activity in which: i) participants would throw

the ball and the robot would fetch it; ii) participants would throw the ball into a basket

shouldered by the robot and compete for the most number of points; and iii) participants would

cooperate and pass the ball among each other in order to keep it away from the robot. The

recreational activity was facilitated by a human coordinator who explained the game to the

residents. The results from the study showed that interacting with the robot and with the other

participants in this recreational activity improved a player’s emotion control and social skills.

In [27], the affective communication robot Matilda was used to facilitate Bingo and Bingo-like

games for nursing home residents in Australia. The robot worked cooperatively with caretakers

in a nursing home to facilitate Bingo and Hoy (a Bingo-clone which replaces the Bingo numbers

with playing cards). Matilda’s task was to call out and display the Bingo numbers and Hoy cards,

while caretakers were responsible for providing one-on-one help to players if they were not able

to hear the numbers, or if they believed they had a winning card. Reactions from the nursing

home residents about Matilda were very positive—residents enjoyed the robot’s body language

and playing the game with the robot.

In multi-user scenarios, the socially assistive robots received positive feedback when they

engaged elderly participants in recreational activities. However, the robots required some human

assistance to engage in multiple interactions with the users throughout the activities.

2.3 Social Human-Robot Interaction

In order for users to adopt a social robot as an assistive aid in the facilitation of recreational

activities, users should display a desire to use the robot and trust in the robot’s advice when it

9

10

performs an assistive behaviour. Intent to use a robotic system is directly correlated with

perceived enjoyment of using the system by the Almere model [28]. As the sociability of a

robotic system has a positive relationship with perceived enjoyment, intent to use is then

positively influenced by how sociable the robot is [28]. Trust in a robot has also been shown to

be influenced by the amount of social intelligence and social abilities in a robotic system [29].

The development of a socially assistive robot should, therefore, take into account types of social

expressions to implement in order to maximize users’ intent to use the robot. Two typical types

of social actions include maintaining eye contact and using arms to gesticulate. The following

sections discuss the use of the non-verbal forms of communication of eye contact and physical

gestures in human-robot interactions.

2.3.1 Eye Contact

Eye contact plays an important role in human communication in directing the flow of

communication and providing cues to help people guess the internal states of the other person

[30],[31]. There have been several research projects which have investigated the role of eye

contact in communication between humans and robots [32]-[34]. One example of a gaze-

communicative robot was presented in [32], which investigated a stuffed toy-like robot that

monitored a user’s gaze direction and performed a gesture or uttered an expression of

acknowledgement whenever eye contact was made. Twenty-two people aged twenty-one to forty

years old were shown the robot placed in between two monitors which showed two animations

randomly as stimuli. They were instructed to look at the robot as it turned its head to look at the

participants and the screens, and then asked what the robot had in mind. One of the primary

conclusions from the study found that the eye contact between the user and the robot provided a

direct evocation of a favorable feeling, and the robot’s gaze was able to draw the attention of a

user to an external stimulus.

In [33], the interactive humanoid robot Robovie was able to display human-like behaviours

through actuators, and vision and audio sensors. In order to test Robovie’s communication with

humans, a version of the robot with eye contact and a version without were given to two groups

of researchers and engineers within the laboratory of the investigators leading the study and were

tasked to obtain the attention of the humans and direct it towards a poster on a wall. The study

found that the participants with the version of the robot with eye contact all indicated that they

10

11

saw a poster, while the majority of the other participants indicated that they saw Robovie’s hand.

The study demonstrated that eye contact was an important factor in obtaining the attention of the

users, as it indicated the robot’s intention to communicate with them.

Another example of eye contact to direct attention during communication between a human and

a robot occurs in [34], when a robot shared joint attention with a human through the use of gaze

tracking. In [34], the expressive humanoid robot Leonardo detected and examined peoples’ faces

in order to determine their expressions as part of a set of imitative behaviours to mimic during

natural social interactions between humans. As part of the face detection, Leonardo also tracked

the user’s eye gaze in order to learn and follow the source of his/her attention. This way,

Leonardo was capable of following the flow of the communication, as well as discovering new

salient aspects of the environment.

2.3.2 Physical Gestures

Robots can use gesturing in order to maintain the attention of a person, explicitly direct

someone’s attention during the flow of communication or express emotion [35]-[39]. The use of

engagement gestures during a human-robot communication interaction can be important to

engage the human and maintain his/her attention [35]. A socially communicative penguin robot

Mel was used to study the efficacy of engagement interactions with human participants in [35].

The robot had two 2 degrees-of-freedom (DOF) shoulders to actuate its wings, a 2 DOF neck to

orient its head, and a 1 DOF beak to open its mouth when speaking. A version of the robot

performed gestures while speaking such as moving its beaks, wings and head, while a second

version of the robot merely used speech and beak movements. 37 participants from the summer

staff at the laboratory ranging from 20 to 50 years of age were split into two groups to interact

with either the fully moving version or the non-moving version of Mel. The participants were

shown a demo of Mel, and then given a questionnaire to investigate their feelings on the robot.

The results found that the participants were more engaged when Mel performed more

movements, and turned their attention to other objects in the environment fewer times during the

demo than the group with the version of the non-gesturing robot. Moreover, participants in the

group where the robot gestured tended to respond to the robot more frequently than the

participants in the other group.

11

12

Arm and head gestures in robots can also play a significant part in displaying the intent of the

robot during a communication interaction, such as when coordinating joint activity between a

robot and a human [36]. A study in [36] investigated the impact of nonverbal communication on

the efficiency and robustness of human-robot teamwork using the humanoid robot Leonardo.

Leonardo was a 65 DOF expressive robot which was capable of orienting its head, moving its

arms to gesture or perform tasks, and shift its body position. Participants from the local

university campus ranging from 20 to 40 years of age were asked to teach Leonardo a task, check

to make sure it knew the procedure to complete the task, and acknowledge when the robot was

able to complete it. During the study, participants relied purely on the non-verbal gestures like

shrugging, nodding, and confusion gestures to diagnose when the robot was not able to

understand instructions. The results of the study demonstrated that having a greater number of

intuitive social gestures significantly increased the speed at which the humans were able to teach

the robot the task. The intuitive social gestures used by Leonardo were able to display the robot’s

intent to participants such that they could immediately adjust their actions in order to better work

with the robot.

Physical pointing in particular is a useful gesture for communicating the intent of a robot towards

a specific focus of attention. In [37], the anthropomorphic upper-torso robot called Nico

performed a series of tasks with participants in order to gauge aspects of social human-robot

interaction such as greetings, cooperation between the user and the robot, and trust in the robot’s

advice. 65 undergraduate students, graduate students and administrative staff at Yale University

participated as subjects in the study. During part of the study, Nico would first wave to a

participant in the room to greet him/her. It would later present a task to the participant by

physically pointing to a stack of books in the room and then to a close-by bookcase to indicate

that the participant needed to move the books from the stack to the bookcase. Nico would only

use this pointing gesture to explain the task. The participants in the room with Nico all correctly

interpreted the implied task from the pointing gesture and moved the books. When Nico then

instructed participants to do the unusual task of throwing out the books by pointing to the stack

of books and to the garbage can, many participants showed some confusion at the task, but still

complied with the request. The study demonstrated that participants could identify the robot’s

intent with only a non-verbal pointing gesture, and generally complied with a request from the

robot even if it was unusual.

12

13

Gestures could also be used to help robots display emotion. The human-like WE-4RII robot in

[38] displayed emotions using facial expressions and upper body movements. The robot

demonstrated emotions including disgust, fear, sadness, happiness, neutrality, surprise and anger.

Each emotion involved a facial expression, head orientation, and arm gestures. When each

emotion was shown to 18 participants with an average age of 21, all of the participants were able

to correctly identify the emotions of surprise, sadness, anger and disgust. The only emotion

which participants had difficulty with was fear, which was many times misidentified as disgust.

The study compared its results with a previous experiment by the same group of researchers in

which a version of the robot without hands demonstrated the same types of emotions to other

participants. The comparison showed that the version of the robot with the complete arms was

able to demonstrate its emotions more effectively to humans, with higher recognition rates across

almost every emotion.

Another example of a robot which incorporated arm gestures when displaying emotions is the

humanoid Nao robot in [39]. The robot generated the emotions of anger, fear, sadness and joy

using the methods of body movements, sounds or eye colors. Nao utilized its entire body

including its arms to display its emotions, and were modeled after analogous human body

movements. Each type of the robot’s methods to display its emotions was demonstrated to 42

participants ranging in age from 19 to 29. Subjects were then asked to identify the emotions

behind the robot’s expressional cues. The results of the study showed that participants

understood the body movements the best out of all the types of emotional cues. The recognition

rates for the emotions behind the body movements indicated that human emotional expression

with body movement was also applicable for expressions with humanoid robots.

2.4 Chapter Summary

This chapter discussed various types of cognitive training programs to help with cognitive and

functional impairments. There are two general types of cognitive training programs, including

specific programs, which target certain cognitive functions, and general programs, which provide

non-specific cognitive stimulation. One type of general training program is recreational

activities, which offer many benefits including cognitive stimulation, social engagement and

health benefits. Bingo is one such recreational activity which can be therapeutically beneficial

for older adults. In order to facilitate Bingo and other recreational activities, social robots have

13

14

been designed as potential activity facilitators. Single user and multi-user social robots have been

designed to facilitate or participate in a wide range of recreational activities. However, many of

these robots require some human intervention during their facilitation of an activity. This thesis

aims to design an autonomous robot which can facilitate a recreational cognitive training activity

without the need for a human operator.

In order to design a socially assistive autonomous robot which can facilitate the recreational

activity Bingo, the social capabilities of the platform must be considered. Two types of non-

verbal social behaviours which can be implemented on a robot are eye contact and physical

gestures. Eye contact has been demonstrated to have many important uses during communication

between humans and robots, including directing the flow of communication and providing cues

to help people guess the robot’s internal states. Gesturing is also important for maintaining the

attention of a person, explicitly directing someone’s attention during the flow of communication

or expressing emotion.

14

15

Chapter 3 Gathering Impressions and Design Considerations

Gather the opinions of end users through a focus group study can aid in the development of a

new technology. This chapter discusses the results of a focus group study done in a long-term

care facility and a retirement home regarding the adoption of Tangy as a recreational activity

facilitator for older adults. The study was done in order to inform the design of various existing

features on Tangy, as well as gain suggestions on features which may be implemented in the near

or long-term future.

3.1 Focus Group Study

Focus group studies were conducted at both a long-term care facility and a retirement home by

our team in order to collect the opinions and concerns of elderly residents, their family members

and healthcare staff about Tangy and other healthcare robots. Written informed consent was

obtained prior to commencement of the study following our Research Ethics Protocol. The focus

groups offered their perspectives on having a socially assistive robot in a long-term

care/retirement home setting, impressions of Tangy from a quick descriptive video of its

capabilities, and suggestions for potential features for Tangy’s design. In this thesis, the

responses were then transcribed and thematically analyzed in order to group common themes or

suggestions across all focus groups.

3.1.1 Description

The long-term care home had recreational programmers who organized regular social activities,

staff to perform housekeeping, laundry and meal preparation duties, and physiotherapists who

guided physical rehabilitation for residents. The long-term care healthcare staff also aided

residents with their activities of daily living, including toileting, eating, dressing, mobility, and

bathing. The retirement home provided residents with housekeeping, laundry and meal

preparation services, as well as regular recreational activities.

There were a total of 16 sessions, each with an average of five participants and lasting

approximately 45 minutes. Each session included exclusively residents, residents’ family

15

16

members, or healthcare workers. Trained moderators from our research group led discussions

according to the set of structured questions listed in Table 1.

Table 1: Focus Group Questions [40]

Focus Group Session Questions 1a) How do you feel about having Tangy assisting residents living at the LTC home? 1b) How do you feel about having Tangy assisting residents living at the retirement home? 2a) What activities do you think Tangy can assist with around the LTC home? 2b) What activities do you think Tangy can assist with around the retirement home? 3a) What specific daily tasks can Tangy personally help you with? (residents) 3b) What specific daily tasks can Tangy help the residents with? (staff/family) 4) How do you feel about having Tangy organize and run Bingo games? 5a) How do you feel about using Tangy to schedule and make video calls to your family and friends? With your doctor? (residents) 5b) How do you feel about using Tangy to schedule and make video calls between residents and their family and friends? With a residents’ doctor? (staff/family) 6a) What features do you think are important in the design of Tangy to make the robot acceptable at the LTC home? 6b) What features do you think are important in the design of Tangy to make the robot acceptable at the retirement home? 7a) What do you see as advantages and disadvantages of having a health-care robot at the LTC home? 7b) What do you see as advantages and disadvantages of having a health-care robot at the retirement home? 8) Do you have any final comments about the robot?

After question 3b) as listed in Table 1 was asked, the moderators showed the participants a

descriptive video of Tangy which depicted it facilitating a Bingo game, and then hosting a

telepresence activity. In the first part of the video, Tangy greeted Bingo players, called out

numbers, and navigated to assist a player with correctly marking his card when he raised his

hand. A segment with Tangy detecting a winning card and celebrating was shown as well. In the

second part of the video, Tangy navigates to an older adult and displays an incoming video call

on its tablet. The user answers the call, and begins having a conversation with the incoming

caller.

16

17

3.1.2 Participant Demographics

Participant information was self-reported using a questionnaire, developed by members of our

research team [40], which asked for gender, age, experience with computers and experience with

robots. Forty residents from the long-term care facility and the retirement facility participated in

the focus group sessions. Residents were only included in the study if they were above the age of

60, spoke fluent English, had good auditory acuity, and had a Cognitive Performance Scale

Score [40] under 3 (indicating mild to no cognitive impairments). Ten family members of

residents participated in the study. Thirty-one healthcare staff from the two facilities (holding

positions including personal service workers, nurses, staff educators, program managers, and

private caregivers) with an average age of 38 participated in the focus groups. Varying levels of

experience with computers were seen across all groups; however, no participants indicated any

experience with robots other than having seen them in museums, science centers, or television

shows. The demographics and experience levels of all participants are displayed in Table 2 and

Table 3, respectively.

Table 2: Participant Demographics

Demographic Average Age (Std Dev.) Female Male

Residents 82.4 (10.3) 33 7 Family Members 66.1 (9.9) 7 3 Healthcare Professionals 38.2 (12.8) 28 3

Table 3: Computer and Robot Experience of Focus Group Participants

Demographic Computer Experience* Robot Experience**

None Beginner Intermediate Advanced None Beginner Intermediate Advanced

Residents 20 13 4 3 33 7 0 0 Family Members 3 1 3 3 8 2 0 0

Healthcare Professionals 2 7 8 14 16 15 0 0

*Beginner (email, use simple programs) Intermediate (internet, chat) Advanced (editing documents, use complex programs) ** Beginner (seen robots at museums/science centers or stores, or have watched shows with robots) Intermediate (have worked with/used commercial robots) Advanced (have worked on robot developmental aspects including hardware/ software design)

17

18

3.1.3 Thematic Analysis

The focus group sessions were audiotaped and later transcribed. The transcriptions were

analyzed with the goal of drawing out thematic sets involving common attitudes, suggestions or

perspectives about the robot and the robot’s role in the long-term care setting. The analysis used

common techniques for decoding focus group results in qualitative research. Participants’

comments were categorized into general impressions, concerns and suggestions about healthcare

robots, or specific impressions, concerns and suggestions for Tangy. Each of these six categories

were then compiled together and qualitatively inspected for similar themes and suggestions. The

thematic sets were then used to guide Tangy’s design.

3.2 Thematic Sets

3.2.1 Bingo

Residents from both facilities indicated that although Bingo was offered regularly at both

facilities, none of them present in the focus groups actually played Bingo. They indicated that the

activity was not particularly interesting to them; however, the residents almost unanimously

stated that they would give Bingo a try if Tangy was there to facilitate the game. Residents also

believed that their friends or neighbours in the facilities would also enjoy participating in the

game with the robot. One elderly participant explained that she would play at least once “from a

curiosity point of view.”

Healthcare staff members thought the robot brought novelty to the Bingo activity, which would

help attract residents to play the game. Some brought up the potential benefits by having a robot

facilitate recreational activities like this, such as lessening the program staff’s workload so that

they have time to create personalized interventions for residents. After showing the descriptive

video of Tangy facilitating a Bingo game for our research team, staff members suggested the use

of alternate methods of interaction between residents and the robot in order to encourage

accessibility, such as the use of leg movements or voice commands. They indicated that the

method of requesting assistance from Tangy demonstrated in the video where participants raised

their hands to get Tangy to come over and help them would present an accessibility barrier to

residents who had impairments with their arms.

18

19

3.2.2 Telepresence

The video demonstration of the telepresence activity was extremely well-received by participants

across all demographics and sessions. Residents were very supportive of the activity as it would

help facilitate communication between them and their family members. They indicated that the

activity gave them freedom so that they wouldn’t have to wait for infrequent visits from their

children and grandchildren. Elderly residents from both facilities gave positive feedback on this

feature due to its ability to facilitate more social interactions between them and their family.

Healthcare staff members indicated that the activity would be very useful and attractive to

residents who had family members who didn’t live close by to the facilities. Staff members also

suggested that the button on the robot’s screen to receive a call be made larger to accommodate

residents with poor eyesight or fine-motor control.

Family members were also largely supportive of the feature and gave similar motives as the

residents. They cited the difficulties in arranging visit times, large distances between them and

the facilities, and their busy schedules as barriers to communication, and noted how the

telepresence activity could overcome these barriers. One family member stated: “I definitely

would use it a lot. I would be using it almost every day.”

3.2.3 Tangy’s Appearance

Opinions about the robot’s appearance varied highly among participant demographics. Elderly

residents in both the long-term care facility and the retirement home were accepting of the

robot’s appearance. Several residents supported the idea that any appearance for Tangy was fine

as long as its physical design was functionality-driven rather than aesthetically driven; one

resident in particular described: “If it looks like a machine, that's fine. I think the most important

thing is what it does.” Several other residents mentioned that Tangy’s machine-like look would

even be more preferable to having the robot look more humanoid. One resident gave her opinion:

“…it could look like a machine, you know…But the idea of having it look like a human being,

which is purposefully done for sure, is wrong, I think.”

Staff participants in both facilities believed that the robot would be more accepted among the

residents if its physical appearance didn’t have as many overtly mechanical features. One staff

member stated that Tangy’s exterior had “too much metal”, and suggested that the robot should

19

20

be made to look “…a little friendlier. If it was furry, or if it looked like an animal or something.

Less machine-like somehow.” One common suggestion among healthcare staff across both the

long-term care facility and the retirement home was to have the robot dressed in everyday

clothing in order to cover up some of its gearing and wiring.

Family members brought up Tangy’s appearance, but were divided on their views about it. Many

participants agreed that the robot’s appearance was agreeable, with some describing: “…he’s got

a very nice expression…”, and they especially liked that Tangy had “…wide, open curious eyes,

and his mouth doesn’t turn down…” However, other family member participants believed the

robot needed to be more human-like in order for residents to be more accepting of it.

3.2.4 Acceptance of the Robot

Some variation of the concern that older adults would not be able to accept the robot because

they had very little experience with similar technologies was brought up in most of the focus

group sessions. Before viewing the descriptive video of Tangy facilitating the Bingo and

telepresence activities, several resident participants indicated that the idea of having a robot in

their facilities scared them; one described: “When you're not used to something, it can be scary.

Certainly at our age, I think.” Nonetheless, several residents gave clear sign that they would be

open to the idea of learning how to use and adopting Tangy in their facilities if its benefits were

clear. One participant stated: “I would say, it'd be non-productive on anybody's part to say they

weren't prepared to look at what the next stage might be…” However, another participant stated

that any desire to adopt a robot may still be tempered by their ability to learn how to use the

technology: “… a learning curve at our age is, generally speaking, not as good as a younger age

like yourselves who have been brought up with all these technologies. That would be another

reaction that I would have, [why] I might be a little bit more resistant to it…”

After being introduced to the robot’s capabilities through the demonstration video and the

moderator’s explanations, the residents were more amenable to having Tangy in the facilities. A

participant who had expressed unease with the robot explained her shift in opinion: “Anything

new, at our age, I tend to look at very warily, but now I feel a great deal more warmth towards a

robot than I ever did before, simply because I've seen it, I understand it, you've explained what

20

21

it's all about.” Similarly, several participants believed that clear explanation of the robot’s

functionality and goal would help alleviate fears about the technology among other residents.

Healthcare staff members brought up one concern about how residents who had cognitive

impairments would react to the robot. They expressed concerns about how moderate to severely

impaired residents may react very aggressively out of fear to something they do not understand.

One participant stated: “[Because] the ones who are stuck in the fifties in their mind, robots were

things way into the future…”, they would not be able to understand the robot or its role in the

facilities.

Similar to the residents and the staff members, family members were unsure whether residents

would be able to understand the purpose or the functionality of the robot.

3.2.5 Ease of Use

An important aspect of the robot’s design among participants was the ease of use of the robot.

Residents required that the technology be easy for them to adopt; one participant expressed: “[the

robot]’s good, as long as it’s easy [to use]…” Moreover, participants pointed out that Tangy’s

benefits would only be relevant if it worked reliably—otherwise, “If it doesn't work, it's a pain.”

Healthcare staff members explained that the usability of the robot would be critical in preventing

feelings of frustration in residents when using it. A staff member pointed out that not only

Tangy’s user interface for residents needed to be clear and intuitive, but also the robot’s expected

behaviours as well: “it will be extremely frustrating to the residents [because] if, here’s

something, and it looks like it’s a machine to help make something easier, and they expect it to

be competent, but it’s not, [then] it’s worse than human beings failing you.” Besides the ease of

use for residents, staff members also wondered about the ease of use for themselves as well:

“What would be the ease of use? ...Would it be really difficult to turn him on and set him up?”

Family members brought up the residents’ lack of knowledge about modern technology again to

highlight the need for the robot to be easy to use.

21

22

3.3 Suggested Features/Activities

3.3.1 Reminders/Prompting

Residents and staff members suggested Tangy be able to give reminders to them and prompt

them about their schedules. Resident participants acknowledged that their memories were being

affected by their age, and gave examples of how they were beginning to forget items on their

schedules. One resident gave an anecdote: “Reminders would be very helpful…I forgot I was

going to get a cup of coffee with a former student yesterday afternoon …I can't imagine doing

that, it's crazy that I should have forgotten that.” Other items suggested for Tangy to remind

residents about included taking medication, responding to missed calls, and going to scheduled

activities.

Healthcare staff members gave similar recommendations for the reminders, and included

prompting during meals and when transporting large groups of residents as a possible feature for

Tangy. Staff participants explained that residents sometimes had difficulties staying focused

during mealtimes or when moving in large groups, and sometimes wandered off. In those

situations, staff members may be focused on another resident or leading a group, so having a

robot to help keeping residents focused would be very helpful.

3.3.2 Simple Interactions with Cognitively Impaired Residents

Healthcare staff and family members believed that having Tangy engage with cognitively

impaired residents would greatly help with their quality of life. Staff participants explained that

they attempted to interact with all residents regularly but sometimes spend less time engaging

with them if they have other critical duties. For residents with cognitive impairments, the social

engagement is simple; any conversation is enough. Participants suggested that Tangy hold

simple conversations with these residents if they were not able to interact with them that day.

Family members similarly pushed for this feature for residents as they believed the social

interaction would stimulate them and help stave off potential loneliness. One participant

described: “They do need that. It stimulates them, they don’t feel so alone, and just sitting there,

no communication, and [the robot] can go around and just talk to them.” Family members

suggested that the conversation need not be complex, as the residents only desired something to

listen to them.

22

23

3.3.3 Other Recreational Activities

As the resident participants in the focus group study were not regular Bingo players, they

suggested a variety of other group recreation activities which were popular at the facilities.

Participants strongly enjoyed trivia and trivia-like games with focuses on history, geography and

grammar. One participant mentioned the benefits of mentally-intensive games like trivia: “you're

asking people to try and recall information, which is another good mental activity. [Which is]

what seniors need. A lot of mental activity.”

3.3.4 Physical Tasks and Aiding Residents’ Autonomy

Having Tangy perform physical tasks in order to help residents and staff members around the

facility was a popular topic of discussion among residents. Resident participants described a

variety of small errands which they or other residents they knew had troubles with: carrying

objects, pushing their wheelchairs around, or helping them out of bed. The only option currently

for residents who needed physical assistance is to call and wait for a staff member or a private

social worker at the facilities. Participants indicated they would rather have a robot help them

rather than wait for a human. One resident also gave another reason that he wanted Tangy to

provide physical assistance: “I always found embarrassing, since I wasn’t in rehabilitation or

anything…There was a time when I couldn't get out of bed by myself. And all I would call the

nurse for was to come and help me out of my bed so I could go to the washroom. Well, that [is

why] I could call a robot, and…that would be acceptable to me.”

3.3.5 Music Therapy

Throughout the study, participants recommended that Tangy incorporate music as a feature.

Residents brought up karaoke and sing-alongs as popular pastimes at the facilities, and suggested

that Tangy be involved in these activities. Staff members mentioned the therapeutic impact of

music, and believed that residents would strongly enjoy having Tangy play music to them.

3.3.6 Multilingual Support

As the long-term care facility and the retirement home had diverse ethnic resident populations,

staff members and family members brought up multilingual support as a feature on Tangy. They

suggested that the robot be able to communicate in other languages both vocally and through text

on its screen in order to connect with residents whose may not be fluent in English. Staff also

23

24

desired Tangy to be able to act as an interpreter between them and residents if there were any

communication gaps.

3.4 Chapter Summary

This chapter investigated the results from a focus group study conducted with three

demographics at a long-term care facility and a retirement home. Residents, family members and

healthcare staff were asked about their general and specific opinions about social robots for the

elderly and Tangy. In general, the three groups of participants believed that Tangy could be

useful in the facilities if it was able to assist with the tasks that were demonstrated in the video

during the study. Participants strongly enjoyed the aspects of the demonstrated activities which

would help facilitate human-human interaction. With respect to Bingo, participants believed that

Tangy would bring an aspect of novelty to the game which would attract more residents to the

weekly games. Healthcare staff in particular identified that the demonstrated method of

requesting assistance from the robot during the game was unfeasible for residents with physical

impairments to their upper body. Participants also brought up the appearance and ease of use of

the robot as important factors to consider when designing the robot to be accepted by the

residents. In particular, all three groups indicated that the robot should have an intuitive interface

and be easy to use, both for residents and the healthcare staff. Finally, participants presented a

variety of suggestions for future features for Tangy, including reminders, prompting, simple

conversations with residents, other recreational activities, physical aid, music therapy and

multilingual support.

24

25

Chapter 4 Robot Behaviours

Tangy’s behaviours are based on the objective of facilitating a Bingo game without the need for

any intervention by a human operator. The robot’s behaviours were designed in order to

encapsulate all of the duties of the human Bingo facilitators at the long-term care and retirement

home facilities in which user studies were planned. From repeated observations of Bingo games

at both facilities, the Bingo facilitator was responsible not only for calling out Bingo numbers,

but also engaging residents with jokes and conversation, assisting players with mistakes, and

celebrating with players when they had a winning card. Tangy’s behaviours imitate the actions of

the human Bingo facilitator by also engaging in social interactions with players by giving jokes

during the game, assisting players one-on-one when they request aid, and performing celebration

dances with players who have winning cards. The robot was developed with five major types of

behaviours corresponding to the various stages of the Bingo game: i) initiating the Bingo

activity; ii) facilitating the game for all players; iii) transitioning from multi-user to single user

behaviours; iv) assisting a single player with his/her card; and v) ending the Bingo activity. In

order to determine which behaviour to implement, Tangy uses a finite state machine. The finite

state machine outputs the appropriate behaviour for the robot given particular trigger events

which change the state of the world, robot, or the players’ cards.

4.1 The Bingo Scenario

A Bingo session begins with Tangy located at the front of the room facing four players seated at

a long table. An hour-long session comprises two or three Bingo games, depending on the

lengths of the games. In front of each player are a Bingo card, markers, and an assistance request

device. Tangy initiates the Bingo session by greeting the residents with a wave and introducing

itself. The robot then explains the rules of the game before beginning to call out the Bingo

numbers. As the robot calls out numbers, players place markers on their card corresponding to

these numbers. Players can request Tangy to come to view their card at any time using their

assistance request device. Tangy is able to offer assistance with incorrectly marked or missing

markers on players’ cards, or award players with victory if it identifies a winning Bingo card.

Tangy concludes calling out numbers when a Bingo game is finished, which occurs either when

one or more players have winning cards, or when all numbers have been called. Depending on

25

26

the game mode, a winning configuration has to have a horizontal, vertical or diagonal line of five

markers on the Bingo board, a fully marked card, or two diagonal lines of five markers.

4.2 System Architecture

Tangy was designed using a multi-layered system architecture, (Figure 2) in order to

autonomously facilitate Bingo games for groups of elderly residents. The architecture consists of

four major layers: the sensory layer, which takes in inputs from the environment; the input

interpretation layer, which transforms the sensory inputs into world, game and robot state

variables; the behaviour determination layer, which uses the state variables in order to determine

the robot’s next actions; and the actuation layer, which transforms the determined behaviours

into outputs that are visible in the environment. Each layer is separated into a modular structure

which helps to organize the different sensory input types.

Figure 2: Tangy's System Architecture

This thesis focuses on the development of the Behaviour Determination layer and the eye

contact, arm control and verbal interactions modules in the Actuation layer. The Behaviour

Determination layer is discussed in the following subsection 4.4 of this chapter. The

development of the actuation layer is also covered in this thesis in the next chapter. In particular,

Chapter 5 discusses the actuation modules of speech, eye contact and arm control.

Behaviour Determination

Actuation State Interpreter Sensing

Face Tracking

Assistance Identification

Bingo Card Detection

Assistive Behaviour Deliberation

Navigation System

Localization

Off-Board IR Sensor

Voice, Screen and

Motors

Laser Range Finder

2D Cameras

Hardware Controllers

26

27

4.2.1 Sensory Capabilities

The sensory layer in the robot architecture employs an array of onboard sensors on Tangy as well

as an off-board sensor in the Bingo activity environment, displayed in Figure 3. An ASUS Xtion

Pro IR sensor is mounted on a stand behind Tangy during a Bingo game in order to capture any

assistance requests from residents. Two sensors help Tangy localize and navigate itself within

the environment: a URG-04LX-UG01 laser range finder captures live data about obstacles in

close proximity to the robot, while optical encoders on Tangy’s wheels continuously record the

distance travelled by each wheel. Tangy’s right eye is a 2D Axis M1031-W camera, and provides

video of users when the robot is assisting them in order to help Tangy track and follow their

faces. Finally, Tangy uses a 2D Logitech Pro C920 webcam mounted on its forehead to provide

high resolution imaging such that it can detect the Bingo card and the card state.

Figure 3: Off-board and On-board Sensors

4.3 Robot States During the Bingo Game

There are five overarching robot states during the Bingo game. The pre-game state occurs before

the game begins, when Tangy gives an introductory speech to the players. The multi-user state

then occurs when Tangy is facilitating Bingo for multiple users at the front of the room. The

transition state defines when Tangy begins to transition in response to an assistance request. The

2D Logitech Pro C920 Camera

2D Axis M1031-W Camera

URG-04LX-UG01 Laser Range Finder

ASUS Xtion Pro IR Sensor

Assistance Request Device

Bingo Markers Bingo Card

27

28

single user state occurs when Tangy is in front of a single user and in the process of assisting

him/her. The final post-game state determines whether the robot concludes the game with a

speech, or continues with a new game. Each state contains a set of pre-determined behaviours,

described in the following sub-sections.

4.3.1 Pre-Game State

The pre-game state is defined before a Bingo game begins, when Tangy gives appropriate

information to players depending on whether this is the first game of the session or a subsequent

game. During the pre-game state, Tangy does not take in any sensory inputs. The behaviours

which Tangy may perform are listed in Table 4.

Table 4: Pre-Game Behaviours

Behaviour Description

Robot Physical Action

Verbal Communication Text on Screen

Introduction Wave “Hi!” “Hi!”

N/A “My name is Tangy! I am so excited to play bingo with you today! It is one of my favorite games.”

“My name is Tangy!”

Instructions Point to screen “I will call a series of numbers, and also display them on my screen. If your card has the same number I called, please mark the number with a red marker.”

“How to play Bingo”

N/A “If you need help with your Bingo card. Or if you think you have Bingo, please press the button on your table.”

“Press button for help”

Starting another game after first

N/A “Let’s play another game of Bingo!”

“Let’s play another game of Bingo!”

N/A “Please clear your cards now.” “Please clear your cards now.”

28

29

4.3.2 Multi-User State

Tangy facilitates the Bingo game for multiple players by calling out numbers and telling jokes

when no assistance requests have been currently detected. The robot selects random numbers

from a vector of strings containing all of the possible Bingo numbers. After a number has been

selected, it is written to a text file for tracking progress while the corresponding string in the

vector is removed. The progress file is used in situations where the robot is unable to continue

playing a game because of a program or server crash. In these situations, a human operator can

restart the Bingo game from the point of failure without having to clear the players’ cards or

restart the introduction speech.

During calling out numbers, Tangy swivels its head to randomly look at pre-set positions where

players would likely be during the game. The head movement allows Tangy to simulate paying

attention to the players from the front of the room by looking at them. The head movement also

provides a more dynamic feel to the robot during the multi-user scenario, when it is mostly

stationary at the front of the room.

Tangy occasionally tells jokes in between calling out numbers. The two-line jokes are stored in a

text file, which the robot reads before beginning the game. After every seventh number is called

out, the robot chooses a random joke from the vector of jokes populated from the original joke

database. After a joke has been told, it is removed from the vector of remaining jokes so as to not

be chosen again. After telling a joke, the robot laughs and brings its hand up to its mouth in a

laugh gesture.

During the multi-user scenario, Tangy constantly scans the environment in order to detect any

assistance requests. The behaviours which Tangy may perform during this state are listed in

Table 5.

29

30

Table 5: Multi-User State Behaviours

Behaviour Description

Robot Physical Action

Verbal Communication Text on Screen

Calling out numbers

Point to screen (every other number)

“The next number is:” “The next number is:”

Head swivel

Bingo Number [Tangy calls out a Bingo number]

Bingo Number [The Bingo number is displayed on the screen]

N/A “One more time” Bingo Number

Head swivel Bingo Number Bingo Number

Telling a joke

N/A Joke set-up (e.g. “What's the best way to carve wood?”)

Joke set-up

N/A Joke punchline (e.g. “Whittle by whittle.”)

Joke punchline

Laugh gesture “Hee hee hee.” Joke punchline

4.3.3 Transition State

Upon detecting an assistance request, Tangy will begin to transition from facilitating the Bingo

game for multiple users to assisting a single user. When an assistance request is detected, the

robot will immediately turn its head towards the location of the request and nod towards the user

in order to acknowledge the request. Tangy will then finish calling out the current number or

state the joke. Subsequently, it will inform the other users that it will help someone and pause the

calling of numbers in order to navigate to the player in need.

30

31

Table 6: Transition Behaviours

Behaviour Description

Robot Physical Action

Verbal Communication

Text on Screen

Acknowledgement of help request

Nod towards location of assistance request.

N/A N/A

Interrupting the game N/A “Please give me a second to help someone.”

"Coming to Help..."

Navigating to the location of the assistance request

Move from front of the room to the assistance request location

N/A "Coming to Help..."

4.3.4 Single User State

Tangy will assist a player, one-on-one if he/she request assistance. If multiple users request

assistance at the same time, or one or more users request assistance when Tangy is currently

helping a user, Tangy will help them in the order that it detects their help requests.

During an assistance sequence with one player, Tangy may move its head in order to look at the

person’s face to establish eye contact. Tangy will attempt to maintain eye contact anytime while

it is interacting with the player. The robot will discontinue eye contact when it is attempting to

view the player’s card.

Tangy identifies any mistakes or winning combinations on a player’s card by viewing it with a

webcam located on the robot’s head. The robot will attempt to look at the card by bowing its

head so its face is angled downwards towards the table in front of it. If it is not able to identify a

card in its field of view, Tangy changes the angle at which its face is bowed two times in order to

capture a field of view closer and further away. The adjustment of its head position is done in

order to ensure that Tangy captures a full range of the view in front of it, so as to not miss seeing

the card.

The list of behaviours Tangy may perform in its single user state is listed in Table 7. How the

robot determines which assistive behaviour to execute is discussed in Section 4.4.

31

32

Table 7: Single user Behaviours

Behaviour Description

Robot Physical Action

Verbal Communication Text on Screen

Greet player Establish eye contact

“Let's check your card to see how you are doing.” OR “Let's take a look at your card.”

“Checking Bingo Card”

Detect card state Bow robot head to look at card (adjust if unsuccessful)

N/A “Checking Bingo Card”

Prompt player to move card into robot’s field of view

Establish eye contact

“I'm sorry. I cannot read your card. Please move it slightly closer and make sure nothing is blocking it.”

“Please Move Card”

Prompt player to fix incorrectly placed markers

Establish eye contact AND Point to screen

“Oops! You have some misplaced markers. Please remove these markers from the following numbers on your card.” OR “Oh jeepers, I think you have some misplaced pieces. Please see if you can remove these markers.” AND List of missing Bingo numbers

“Please place markers on:” AND List of missing Bingo numbers

Prompt player to mark missing numbers

Establish eye contact AND Point to screen

“Oh no! You have some missing markers. Please place markers on the following numbers on your card.” OR “I think you have some missing numbers. Can you place markers on the following spots?” AND List of missing Bingo numbers

“Please place markers on:" AND List of missing Bingo numbers

32

33

Praise player and then encourage him/her

Establish eye contact

“You're doing amazing!” OR “You must have played this before!” OR “Your card looks great!” OR “Wow, you are close to getting Bingo!”

“GREAT JOB!”

N/A “Keep up the good work!” OR “You're almost there!” OR “Just a few more numbers before you're there!”

“GREAT JOB!”

Congratulate player on a winning card

Establish eye contact AND Do celebration dance

“Congratulations! You have won Bingo!”

"BINGO!!!!!!!!"

4.3.5 Post-Game State

The post-game state is similar to the pre-game state, where Tangy takes in no sensory inputs and

states a few farewell statements. Tangy’s possible behaviours during the post-game state are

given in Table 8.

Table 8: Post-Game Behaviour

Behaviour Description

Robot Physical Action

Verbal Communication Text on Screen

Give farewell

N/A “Well that's it! Wasn't that fun? Thank you very much for playing Bingo with me. I had so much fun!”

“Thank You!”

Wave “Goodbye!” “Goodbye!”

4.4 Behaviour Determination

Robot behaviours are determined using the robot, card, and world states. The robot states govern

the flow of the Bingo game and are discussed in Section 4.2. The card states are only used when

33

34

Tangy is assisting a single user with his/her card, and determine the specific assistive behaviour

discussed in Section 4.3.4. The world state determines when Tangy enters the transition state

described in Section 4.3.3, as well as how the robot navigates through the environment. The

various states and their possible values are listed in Table 9.

Table 9: Behaviour-Determining States

Type of State Value

Robot State

Pre-Game State

Multi-User State

Transition State

Single user State

Post-Game State

Card State

Occluded

Correctly marked

Missing number(s)

Incorrectly marked number(s)

Winning combination

World State

Location of assistance requests in environment

Obstacles in the environment

Location of robot in the environment

Tangy, as the game facilitator, utilizes a finite state machine (FSM) in order to choose which

behaviours to implement. The FSM was developed so that it was structured around the flow of a

typical Bingo game. Namely, the robot chooses and executes its behaviours based on the

progression of the game. The progression of a Bingo game requires the following primary events

to occur: i) the game begins; and ii) Bingo numbers are consecutively presented to players.

Secondary, player-dependent events may also occur if: i) a player indicates he/she requires

assistance during the game; or ii) a player claims to have a winning card and informs the game

facilitator. With respect to the primary Bingo events, the Bingo facilitator is responsible for

initiating the game and calling out numbers. With the secondary events, the facilitator is

responsible for aiding a player with questions about the game or about his/her card, and either

34

35

accepting or declining a claim for a winning card. Any other responsibilities, such as social

interactions between the facilitator and players, or post-game actions are supplementary to the

facilitator’s core behaviours.

The FSM implements Tangy’s behaviours required for the primary events before all others.

Namely, the robot executes the pre-game and multi-user state behaviours described in sections

4.2.1 and 4.2.2. The FSM only implements the secondary behaviours if there is a world state

change where an assistance request has been located in the environment. These behaviours

include the transition and single user state behaviours described in sections 4.2.3 and 4.2.4. The

post-game behaviour in section 4.2.5 is supplementary, and occurs after the primary and

secondary behaviours have been executed.

Tangy’s FSM is presented in Figure 4, in which the circles represent the individual behaviours

and the arrows represent the behaviour transitions which would trigger the consequent behaviour

to execute. Each behaviour is color-coded in order to categorize them based on the robot state

that the behaviour belongs in.

35

36

Figure 4: Tangy’s FSM – Pre- and post-game states are in light blue; multi-user state behaviours are in green; transition actions are in grey; and the individualized help behaviours are in pink.

Every Bingo session begins with Tangy in the pre-game state. Upon completion of its

introduction and instruction behaviours, Tangy will then begin to facilitate the game in its multi-

user state.

When facilitating the game for multiple users, Tangy will continuously execute its calling out

Bingo number and telling joke behaviours. The joke behaviour is executed after every seven

Bingo numbers have been called. The robot will only progress from these behaviours upon a

detection of an assistance request in the environment.

36

37

After detecting an assisting request, Tangy will perform an acknowledgement behaviour

immediately, and navigate to the location of the request only after it finishes calling out the

current number.

Upon reaching the first player who requested assistance, the robot will then immediately perform

its single user greeting behaviour, and then progress to attempt card detection. If the occluded

card state occurs, then the robot will prompt the player to move the card into its field of view.

Otherwise, if the incorrectly marked card state occurs, the robot will prompt the player to fix the

incorrectly placed markers. Then, if the missing markers card state occurs, the robot will prompt

the player to place markers on the missing spots. The robot will execute these three behaviours

repeatedly until the card state becomes correctly marked. If at this point the card does not contain

a winning combination, Tangy executes its praise and encouragement behaviour. If the card does

contain a winning combination, Tangy executes its celebration behaviour.

After assisting one person, Tangy will re-enter the transition state until all assistance requests

have been fulfilled. Should a winning card not been identified after every player has been helped,

Tangy will re-enter the multi-user state and begin calling out numbers again. Otherwise, Tangy

will enter its post-game state and begin giving its farewell behaviour.

Figure 5: Tangy Behaviours during Bingo Game: (a) Greeting and Introduction at the beginning of a game; (b) Calling out and pointing to Bingo numbers displayed on its screen; (c) Navigating to player; (d) Giving corrective assistance aurally and visually through its screen; (e) Celebrating a winning card by dancing with its arms swaying from side to side in the air; (f) Saying and waving goodbye at the end of a session.

37

38

4.5 Chapter Summary

This chapter presented the development of the robot behaviours and behaviour determination

technique for Tangy to autonomously facilitate the Bingo scenario. The robot behaviours were

designed to imitate the behaviours of a human Bingo facilitator, and encapsulate both the multi-

user and single user interactions seen in a typical Bingo game in a long-term care facility and

retirement home. Tangy’s behaviours fell within five major categories: i) pre-game greeting and

instructions; ii) multi-user interactions; iii) transition actions; iv) single user assistive behaviours;

and v) post-game farewell. For each behaviour, the robot performs some combination of a

physical action, a verbal auditory statement, and a visual display of a text statement on its tablet

screen. Tangy determines its behaviours using an FSM. The FSM outputs a behaviour based on

triggering events which occur when one of the world, robot or card states change.

38

39

Chapter 5 Robot Actuation Modules

The following sections discuss the development of the arm control, eye contact and audio

interactions modules.

5.1 Arm Control Module

Tangy displays body language primarily through arm gestures and head movements. The arm

gestures currently implemented include waving to players, pointing to the robot’s onboard

screen, covering the robot’s face when laughing, and swaying its arms above its head when

celebrating a winning card. Tangy’s gestures are used to draw players’ attention to important

information on its screen, provide natural non-verbal communication common in social

interactions, and amuse players.

5.1.1 Motion Planning for the Arms

Tangy implements ROS MoveIt! [42] in order to create motion trajectories for the arm gestures.

Motion plans consist of a set of joint angles for a set kinematic chain over a trajectory from an

initial position to a final position. MoveIt! generates point-to-point motion plans for kinematic

chains dynamically using motion planners in the Open Motion Planning Library [43]. The real-

time generation of the plans provides unique trajectories for the arms, in order to have Tangy’s

arm gestures seem less mechanical and machine-like. Moreover, dynamically planning the

gestures allows for the option of generating obstacle-avoiding trajectories.

In order to generate plans for Tangy’s arm gestures, a 3D assembly of the robot was first created

in the solid modelling computer aided design (CAD) software SolidWorks in order to establish

the kinematic chains for the planner. The models of the arm linkages were created according to

the measured dimensions of each link in Tangy’s arms. The robot’s upper torso, and the tablet on

the front of the robot, was also modeled using the dimensions of the actual robot torso and tablet,

as it was important to properly define the workspace of the robot’s arms. The joints of the

modelled robot were constrained to rotate about the same axes as the analogous joints on the

physical robot, including the pitch and roll capabilities of the 2 DOF neck, the pitch, yaw and roll

DOF shoulder, the pitch DOF elbow, and the pitch and roll capabilities of the 2 DOF wrist. A

picture of the model is displayed in Figure 6.

39

40

Figure 6: CAD Model of Tangy and Kinematic Model of Tangy’s Arm The CAD model of the robot was imported into the MoveIt!-specific Unified Robot Description

Format (URDF) using the SolidWorks to URDF exporter plugin for SolidWorks, which

automatically generated a series of physical properties for all of the parts of the modelled robot.

The URDF file defines the robot model in a large XML format file, with linkage dimensions,

physical and collision properties, and joint movement limitations defined through pre-set fields.

The URDF file is also used to set up key configuration files for Tangy in MoveIt!, such as the list

of the kinematic chains on the robot available to the planners, the initial poses for the kinematic

chains, and the collision matrices of each of the robot’s parts which is calculated from their

dimensions and physical properties.

The arm motion plans are generated using a serial kinematic chain, a starting pose, and an end

pose as initial inputs. A set of ten plans is always generated in tandem for each gesture, from

which an optimal plan with the smallest time duration and the smallest battery discharge amount

is chosen. The optimal criteria were chosen to provide the most responsive gesture while

preserving the robot’s battery levels. The optimizer calculates time duration based on timestamps

native to each point in motion plans, and energy consumption based on the Reverse Newton-

Euler Algorithm [44] used in inverse dynamics for kinematic chains.

40

41

5.1.2 Optimization Parameters

The two parameters of the motion plans chosen to be optimized were time and energy

consumption of the implemented plans. Motion plans with shorter time durations have fewer

points within the plans, which generally result in fewer inefficient movements through the

gesture. The time parameter for a motion plan for one of Tangy’s arm movements is defined to

be the amount of time the robot takes to move from an initial arm pose to a final arm pose. The

duration of a plan is calculated using the duration field specified in each point in an arm motion

plan class object. The total duration of the plan is equal to the sum of the durations of each point

in the motion plan.

𝐷𝐷𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = � 𝐷𝐷𝑝𝑝𝑝𝑝

# 𝑃𝑃𝑝𝑝𝑃𝑃.𝑖𝑖𝑝𝑝 𝑃𝑃𝑝𝑝𝑝𝑝𝑝𝑝

𝑖𝑖=0

(1)

The estimated energy consumption was chosen to be optimized in order to minimize battery

usage and increase the amount of Bingo games the robot can facilitate one session. As the

robot’s primary power drains include arm movements, head movements, navigation with the

mobile base, and idling all of the servos and sensors, reducing the amount of energy required for

performing the gestures during a Bingo game could result in measurable differences in runtime

on a single charge of the battery.

The optimization approach determines the energy scores of a motion plan by first solving the

inverse dynamics problem for the kinematic chain for each point to point movement in the plan.

The Recursive Newton-Euler Algorithm is used to solve for the torques at the joints. Using

forward recursion from the first link (shown in Figure 6 in the first coordinate frame of the

kinematic model) in the robot’s arm to the last, the angular velocity ωi, the angular αi and linear

accelerations ai of each link i can be obtained, assuming that the base link does not move and

initial linear and angular velocities and accelerations are 0:

41

42

𝜔𝜔𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝜔𝜔𝑖𝑖−1 + 𝑧𝑧𝑖𝑖 ∙ 𝜃𝜃�̇�𝚤

(2)

𝛼𝛼𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝛼𝛼𝑖𝑖−1 + 𝑧𝑧𝑖𝑖 ∙ �̈�𝜃𝑖𝑖 + 𝜔𝜔 × 𝑧𝑧𝑖𝑖 ∙ 𝜃𝜃�̇�𝚤 (3)

𝑎𝑎𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝑎𝑎𝑖𝑖−1 + �̇�𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1 + 𝜔𝜔𝑖𝑖 × (𝜔𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1) (4)

𝑎𝑎𝑐𝑐𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖𝑖𝑖−1 ∙ 𝑎𝑎𝑖𝑖−1 + �̇�𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖 + 𝜔𝜔𝑖𝑖 × (𝜔𝜔𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖) (5)

Backwards recursion can then be applied in order to find the forces and torques on the joints of

the robot. Using the results from equations (2) to (5), the forces fi and torques τi can be calculated

working from the last link to the first:

𝑓𝑓𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖+1𝑖𝑖 ∙ 𝑓𝑓𝑖𝑖+1 + 𝑚𝑚𝑖𝑖 ∙ (𝑎𝑎𝑐𝑐𝑖𝑖 − 𝑹𝑹𝑇𝑇 ∙ 𝑔𝑔)𝒊𝒊

0 (6)

𝜏𝜏𝑖𝑖 = 𝑹𝑹𝑇𝑇𝑖𝑖+1𝑖𝑖 ∙ 𝜏𝜏𝑖𝑖+1 − 𝑓𝑓𝑖𝑖 × 𝑟𝑟𝑖𝑖−1,𝑐𝑐𝑖𝑖 + 𝑹𝑹𝑇𝑇𝑖𝑖+1

𝑖𝑖 ∙ 𝑓𝑓𝑖𝑖+1 × 𝑟𝑟𝑖𝑖,𝑐𝑐𝑖𝑖 + 𝜔𝜔𝑖𝑖 × (𝐼𝐼𝑖𝑖 ∙ 𝜔𝜔𝑖𝑖) + 𝐼𝐼𝑖𝑖 ∙ 𝑎𝑎𝑖𝑖 (7)

Subsequently, the optimization technique would then take the vector of torques τ obtained for

each series of movements and find the energy required for each set of movements from one point

to another through a motion plan. The power P for each movement increment is found by

multiplying the torques τ by the angular velocity �̇�𝜃. The energy for each movement E is then the

power multiplied by the duration of the increment dt between the points. The energy for an entire

plan for one joint Ejoint is the sum of all of the energies for each point in the plan for that joint:

𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 = � 𝑃𝑃(𝑡𝑡)𝑑𝑑𝑡𝑡𝑝𝑝

0= � (𝑃𝑃𝑖𝑖(𝑖𝑖) ∗ 𝑑𝑑𝑡𝑡)

𝑝𝑝

𝑖𝑖=1

𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 = � �𝜏𝜏𝑖𝑖 ∗ 𝜃𝜃�̇�𝚤(𝑖𝑖) ∗ 𝑑𝑑𝑡𝑡�𝑝𝑝

𝑖𝑖=1

(8)

However, the energy required to move the arm in the motion plan must be adjusted for the type

of servos in each joint in order to accurately estimate the energy required to execute the plan.

Tangy’s manufacturer, Dr. Robot Inc., supplied the servo model numbers for each of the joints.

The servo models and their operating energy consumption values are listed in Table 10. The

operating power consumption of each joint was used to adjust the sum in equation (8) by

weighting the energies according to the type of servo at the joint. The weighting is done through

multiplication of a weighting factor Wi.

42

43

𝐸𝐸𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 = � 𝐸𝐸𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝 𝑖𝑖 ∗ 𝑊𝑊𝑖𝑖

# 𝑗𝑗𝑗𝑗𝑖𝑖𝑝𝑝𝑝𝑝𝑃𝑃

𝑖𝑖=1

(9)

The weighting factor is calculated by normalizing the power consumptions for the joints so that

the servo with the highest power consumption has a weight of 1.

Table 10: Manufacturer Servo Specifications for Tangy's Arms [45]

Joint: Servo Model: Operating Power Consumption: Shoulder Roll Joint HS-805BB 830 mA @ 6.0V = 4.98 W Shoulder Pitch Joint HS-805BB 830 mA @ 6.0V = 4.98 W Shoulder Yaw Joint HS-785HB 285 mA @ 6.0V = 1.71 W

Elbow HS-785HB 285 mA @ 6.0V = 1.71 W Wrist Pitch HS-645MG 350 mA @ 4.8V = 1.68 W Wrist Roll HS-645MG 350 mA @ 4.8V = 1.68 W

The techniques to calculate duration and energy costs used in the simulation described in the

previous section were then implemented on Tangy using the following technique. The

optimization approach on the robot takes in ten plans from the motion planner, and calculates the

duration costs for each. If the differences between the lowest duration plan and subsequent

plan(s) are within 0.25 seconds of each other (which is visually very similar to one another), then

the optimization approach takes all of these similar plans and calculates the energy costs for

each. The optimization approach searches for the lowest duration plan first as gestures which

take longer to execute than needed may look unnatural either due to its slow speed or extraneous

movements. After the energy costs have been calculated for each plan, the plan with the lowest

energy cost is finally chosen to be executed.

5.2 Eye Contact Module

In order to provide eye contact as a social behaviour for players when assisting them in single

user scenarios, Tangy tracks the location of a user’s face and attempts to follow that location

with its head in order to make and maintain eye contact with him. This feature was implemented

in order to mimic the eye contact that residents would receive when requesting one-on-one

assistance from a human facilitator.

43

44

The eye contact module uses a video stream from the 2D Axis M1031-W camera in Tangy’s

right eye to scan for and track faces in the robot’s field of view. The OKAOTM Vision software

library [44] searches for facial features within 2D images from the video stream in order to

identify players’ faces (as displayed in Figure 8). The module is rotation invariant within the

following ranges: 30°, 20° and 360° in the yaw, pitch or roll directions (Figure 7). The location

of every face within the field of view of the 2D video stream is obtained as a set of pixel

coordinates with respect to the horizontal and vertical axes. The coordinates are defined by an

origin at the top left hand corner of the video stream, and increasing X and Y coordinates going

downwards and to the right respectively. The center of the 2D image is considered to be the

direct line-of-sight; as the resolution of the video feed is 640 x 480 pixels, the direct line-of-sight

is considered to be located at the coordinates (320, 240).

Figure 7: Face Detection Rotation Invariance Tangy identifies the person closest to its direct line-of-sight as the “person of interest”, and

assumes that this is the player who requested assistance. If the location of the person of interest’s

face is offset from center of the robot’s direct line-of-sight, Tangy will actuate its neck servos in

the pitch and roll axes in order to reorient its line-of-sight. The specific servo commands sent to

the neck are converted from the offset distances in pixels through an empirically tested

conversion technique. The conversion technique sends a maximum servo increment if the offset

distance value is over 150 pixels, and a parameterized servo increment (c * offset_value) if the

offset is under 150 pixels. The parameter c was chosen to be 0.4, after several tests with the eye

contact module. The parameter was chosen in order to limit the movement of Tangy’s head when

44

45

attempting to make eye contact with a player to speeds which are similar or slightly slower to a

human making eye contact with another human. Tangy continuously sends incremental servo

commands until the offset distance value approaches 0, allowing the robot to maintain unbroken

eye contact.

Figure 8: Face Tracking

5.3 Audio Interactions

Tangy employs verbal communication and music to interact aurally with residents. During the

multi-user aspect of the Bingo activity, the robot plays ragtime music in the background in order

to keep residents engaged during moments of silence between numbers. At the same time, Tangy

also continuously calls out Bingo numbers and tells jokes. During the single user aspect of the

Bingo activity, Tangy pauses the music in order to provide clear verbal instructions to players

when assisting them. Both the verbal communication and music are handled by the audio

interactions module.

In order to provide the verbal communication, Tangy uses a pitch-adjusted female voice

synthesized by GoogleTM powered text-to-speech. The background songs are stored and played

at random continuously during the game when Tangy is calling out Bingo numbers. All audio

files are played using an audio player developed using the Simple DirectMedia Layer library

[47].

5.4 Chapter Summary

This chapter discussed the eye contact module, the arm control scheme and optimization

technique, and the audio interactions module. Tangy controls its arms using a set of motion

(246,138)

45

46

planners in the Open Motion Planning Library, which generate point-to-point plans that take a

kinematic chain from an initial to a final pose. An optimization technique was developed in order

to minimize both the duration of arm movements during the execution of a gesture, and their

energy consumption as well. The optimization was done in order to avoid unnaturally slow arm

movement speed or extraneous movements, and to increase the runtime of the robot on a single

charge of battery.

Tangy initiates eye contact with a Bingo player by first tracking his/her face in a 2D video

stream obtained by a webcam in its eye. The robot then maintains eye contact by sending servo

commands to its neck to reorient its own face to decrease the offset distance between the player’s

face and its direct line-of-sight.

Finally, Tangy uses a speech synthesizer to provide verbal interactions with players. The robot

also plays background music to engage residents during silences in between calling out numbers.

46

47

Chapter 6 System Performance Review

To evaluate the robotic system, the finite state machine and the arm control were tested internally

in the research lab. The performance review for the finite state machine included playing the

equivalent of 50 Bingo games with five players and recording the success rates of behaviours.

The performance review for the arm control included two separate experiments; the first tested

the accuracy of the arms, and the second tested the optimization technique to minimize duration

of execution and energy consumption when performing a gesture.

6.1 Finite State Machine Performance Review

The finite state machine was tested in tandem with the sensory capabilities of the robot. The

performance review was done in order to verify that the robot was able to choose and execute its

behaviours appropriately when the world, robot, or card state changes.

6.1.1 Methods

In order to test the performance of the robotic system, the robot played the equivalent of 50

Bingo games by performing its multi-user, transition and single-user interactions repeatedly.

Each repetition of the game involved the robot calling out numbers and telling jokes,

transitioning to assist with an incorrectly played Bingo card, providing appropriate assistive

instructions until the card has been fixed, and continuing the game until a winning card has been

identified. During Tangy’s implementation of its multi-user interaction, the instances of the

Bingo number-calling behaviour and the joke-telling behaviour were counted to ensure that they

were occurring with the right frequency. Each time an assistance request device was activated,

the robot was monitored to identify the success rate of properly acknowledging the request and

then properly transitioning to the location of the assistance request. During the single-user

interaction, Tangy was tested to investigate whether it was able to identify cards with both

incorrectly marked and missing markers, and whether it was able to provide the appropriate

instructions to correct the card. The robot was also tested to see whether it would properly

identify correctly marked cards or cards with winning combinations, and whether it would

consequently provide the praise and encouragement or celebration behaviours respectively.

47

48

6.1.2 Results

The finite state machine performance results are compiled in Table 11. The results demonstrate

that Tangy was able to successfully choose the correct behaviour provided that a state change

was properly identified and sent to the finite state machine as an input.

Table 11: Finite State Machine Performance

State Change Input Expected Robot Behaviour Success Rate (%)

Playing Bingo (no state changes) Robot calls out random Bingo numbers from 1 to 75 100

7 Bingo Numbers Called Robot tells a joke or a trivia fact about Bingo 100 Help Request Detected in the

World Robot acknowledges assistance request by nodding its

head towards the user 100

Help Request Location Detected; Robot Location

Changes Robot navigates to a user in order to provide help 100

Bingo Card State Detected Robot: 1) instructs user to unmark incorrect

numbers/mark missing numbers; 2) provides praise; or 3) congratulates user and does a celebration dance.

100

6.2 Arm Control Accuracy Experiment

The arm control system was tested by moving individual joints at a time to investigate the

accuracy of the movements being executed by the arm servos. The experiment sought to test the

hardware accuracy of Tangy’s arms by taking measurements of the pose of the arm with an

external sensor after given a pre-determined goal pose.

6.2.1 Methods

The accuracy of the arm servos was tested for each joint in both of Tangy’s arms (Figure 9), with

the exception of the finger joints. The experiment procedure consisted of sending a goal rotation

of a certain angle for each joint and then measuring the actual movement of the arm to compare

the actual rotation of the joint with the goal. In order to simplify the analysis, each joint was

tested independently so as to more easily obtain the angle difference from the initial to the final

position. The finger joints were not considered as they are not used in any of the gestures

implemented on Tangy.

48

49

Figure 9: Servo Number Scheme (Right Arm) [48]

The external sensor used to measure the angle changes from the rotation of Tangy’s joints was a

three-axis accelerometer. The Texas InstrumentsTM CC2650 SensorTag, which contains a suite

of various sensors including a three-axis accelerometer, was specifically chosen for this

experiment for its cheap cost, small profile and its pre-established data collection interface. The

SensorTag measures 5.1cm x 3.7cm x 0.9cm, and connects to a smartphone app through a

BluetoothTM connection. The smartphone app can automatically upload data onto a cloud-based

storage system. The SensorTag’s accelerometer is the MPU-9250 model from sensor

manufacturer InvenSenseTM. The MPU-9250 has a ±60 mg tolerance in the X and Y axes at rest,

and a ±80 mg tolerance in the Z axis.

49

50

Figure 10: Texas Instruments CC2650 SensorTag

In order to convert accelerometer readings to angle measurements, simple vector geometry is

used. If the accelerometer readings for gravity in frames of the starting arm pose and final arm

pose are 𝒂𝒂𝟏𝟏 = �𝑎𝑎𝑥𝑥1,𝑎𝑎𝑦𝑦1,𝑎𝑎𝑧𝑧1� and 𝒂𝒂𝟐𝟐 = �𝑎𝑎𝑥𝑥2,𝑎𝑎𝑦𝑦2,𝑎𝑎𝑧𝑧2� respectively, then the difference in angle

between the gravity vectors in the two frames can be found from the dot product of the two

vectors:

𝒂𝒂𝟏𝟏 ∙ 𝒂𝒂𝟐𝟐 = |𝒂𝒂𝟏𝟏||𝒂𝒂𝟐𝟐| ∗ cos (∆𝜃𝜃) (10)

∆𝜃𝜃 = cos−1

⎝

⎛ 𝑎𝑎𝑥𝑥1 ∗ 𝑎𝑎𝑥𝑥2 + 𝑎𝑎𝑦𝑦1 ∗ 𝑎𝑎𝑦𝑦2 + 𝑎𝑎𝑧𝑧1 ∗ 𝑎𝑎𝑧𝑧2

�𝑎𝑎𝑥𝑥12 + 𝑎𝑎𝑦𝑦12 + 𝑎𝑎𝑧𝑧12 ∗ �𝑎𝑎𝑥𝑥22 + 𝑎𝑎𝑦𝑦22 + 𝑎𝑎𝑧𝑧22 ⎠

⎞ (11)

In this experiment, the joints are only given rotational goals about the major X, Y, and Z axes of

the world frame. The rotational goals which Joints 4-8 are given are all 90°. The rotation goal

given for Joint 3 is 45°, due to the fact that the joint is limited to a hardware range of ±45°. The

resulting poses from the rotation of the joints is illustrated for the right arm in Table 12, and is

mirrored for the left arm. The initial starting position for the test of Joints 3, 5, 7, and 8 will be

the neutral position. The starting position for the test of Joints 4 and 6 will be Pose 2 and 5

respectively, as the accelerometer is not capable of detecting roll about the vertical Z axis in the

world frame.

The performance review investigated the error between a measured rotation and the goal rotation

for each joint.

50

51

Table 12: Robot Right Arm Poses

Pose Number Pose Description Pose Picture

1 Neutral Position – All joints are at 0°

2 Joint 8 (shoulder roll joint) is rotated 90°

3 Joint 7 (shoulder pitch joint) is rotated 90°

4 Joint 6 (shoulder yaw joint) and Joint 8 are rotated 90°

5 Joint 5 (elbow joint) is rotated 90°

51

52

6 Joint 4 (wrist roll joint) and Joint 5 are rotated 90°

7 Joint 3 (wrist pitch joint) is rotated 45°

6.2.2 Results and Discussion

The results from the accuracy test for the right and left arms are listed in Table 13 and Table 14.

The mean errors for each joint in both arms are listed in Table 15. In general, the accuracy of the

movements of all of the joints in the right arm was within 5% of the goal pose, while the

accuracy of the left arm was within 8%. The errors of the joint movements for both arms could

be attributed to noise when attempting to measure the pose of the arm with the external sensor;

Tangy’s arms vibrate slightly, which resulted in a large range of accelerometer readings when

testing the right arm. The accelerometer values in Table 13 are the mean of the readings taken

from 10 consecutive accelerometer readings.

The performance review demonstrated that Tangy’s right arm was more accurate when executing

a goal pose than the left arm. These results suggest that any gestures which require any level of

precision, such as a pointing gesture, should be implemented on Tangy’s right arm rather than its

left arm.

52

53

Table 13: Accuracy Performance Review for Tangy's Right Arm

Trial 1

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 0.01 0.97 0.02 0 -

8 1 2 0.96 0.03 -0.03 87.6 3%

7 1 3 0.04 -0.01 0.96 89.4 1%

6 2 4 0.02 -0.04 0.99 91.1 1%

5 1 5 -0.02 -0.01 0.97 89.4 1%

4 5 6 0.98 0.01 0.04 88.8 1%

3 1 7 -0.01 0.683 0.685 43.9 2%

Trial 2

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 0.01 0.97 0.02 0 -

8 1 2 0.96 0.03 -0.03 87.6 3%

7 1 3 0.04 -0.01 0.96 89.4 1%

6 2 4 0.02 -0.04 0.99 91.1 1%

5 1 5 -0.02 -0.01 0.97 89.4 1%

4 5 6 0.98 0.01 0.04 88.8 1%

3 1 7 -0.01 0.683 0.685 43.9 2%

Trial 3

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 0.01 0.97 0.02 0 -

8 1 2 0.96 0.03 -0.03 87.6 3%

7 1 3 0.04 -0.01 0.96 89.4 1%

6 2 4 0.02 -0.04 0.99 91.1 1%

5 1 5 -0.02 -0.01 0.97 89.4 1%

4 5 6 0.98 0.01 0.04 88.8 1%

3 1 7 -0.01 0.683 0.685 43.9 2%

53

54

Table 14: Accuracy Performance Review for Tangy's Left Arm

Trial 1

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 -0.1 1.04 -0.01 0 0

8 1 2 -0.92 0.03 0.01 82.6 8%

7 1 3 0.1 0.06 -0.92 86.3 4%

6 2 4 -0.07 -0.02 -0.99 90.2 0%

5 1 5 0.05 0.07 -0.93 85.4 5%

4 5 6 -0.92 -0.09 0.1 90.1 0%

3 1 7 0.04 0.707 0.707 46.1 3%

Trial 2

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 0.04 0.99 0.03 0 0

8 1 2 -0.92 -0.07 -0.13 96.8 8%

7 1 3 -0.01 0.03 -0.93 89.9 0%

6 2 4 0.09 0.09 -0.95 86.1 4%

5 1 5 -0.04 0.03 -0.95 90.0 0%

4 5 6 -1.01 -0.04 -0.09 94.7 5%

3 1 7 0.05 0.693 0.696 43.3 4%

Trial 3

Testing Joint

Initial Pose

Final Pose

Accelerometer X Axis (g’s)

Accelerometer Y Axis (g’s)

Accelerometer Z Axis (g’s)

Δθ (Degrees) % Error

- 1 1 0.1 0.96 -0.09 0 0

8 1 2 -0.91 -0.04 0.01 98.4 9%

7 1 3 -0.09 -0.01 -1 85.7 5%

6 2 4 0 -0.08 -0.97 89.3 1%

5 1 5 0.05 0.02 -0.9 83.0 8%

4 5 6 -1.02 -0.02 0.09 97.4 8%

3 1 7 0.1 0.75 0.66 46.4 3%

54

55

Table 15: Mean Errors in Joint Accuracy Performance over Three Trials

Joint Right Arm Mean Errors

Left Arm Mean Errors

8 3% 8% 7 1% 3% 6 3% 2% 5 2% 4% 4 2% 5% 3 4% 3%

6.3 Arm Optimization Approach Experiment

The optimization technique in the arm control was tested by comparing the measured duration

and energy consumption of the optimized and un-optimized execution of various gestures. The

experiment tested the wave, point to screen, laugh, and celebration gestures to ensure that the

optimization approach was effective in optimizing different lengths of plans and ranges of

motion.

6.3.1 Methods

Tangy performed the wave, point to screen, laugh and celebration gestures each 25 times in a

trial which implemented the optimization technique. The robot planned a new trajectory after

each execution of a gesture, with a total of 100 unique plans created and executed during one

optimized trial. Then, Tangy performed the same procedure with the gestures in a trial which did

not optimize for time or energy consumption. The durations and energy consumption readings

for the optimized and un-optimized cases were taken for each group of repetitions of gestures. A

baseline energy consumption reading when the robot remained idle was taken for each gesture to

establish the amount of energy consumed by other power drains such as the robot base, neck

servos and the laptop. In order to measure the energy consumption during the trials, voltage

readings from Tangy’s battery were taken before and after each set of repetitions of gestures. The

energy consumption of the gestures will be defined as by the depth of discharge percentage

value. The depth of discharge is calculated by determining the initial and final states of charge of

the battery when compared to a theoretical full and empty charge (36 V and 25.9 V respectively,

as according to the manufacturer’s standards [48]). The equation to calculate the depth of

55

56

discharge D.O.D. from the initial battery reading Vinit, final battery reading Vfin, theoretical full

charge Vfull and theoretical empty charge Vempty is as follows:

𝐷𝐷.𝑂𝑂.𝐷𝐷. =𝑉𝑉𝑖𝑖𝑝𝑝𝑖𝑖𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦𝑉𝑉𝑓𝑓𝑓𝑓𝑝𝑝𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦

−𝑉𝑉𝑓𝑓𝑖𝑖𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦𝑉𝑉𝑓𝑓𝑓𝑓𝑝𝑝𝑝𝑝 − 𝑉𝑉𝑒𝑒𝑒𝑒𝑝𝑝𝑝𝑝𝑦𝑦

(12)

6.3.2 Results and Discussions

The results from the two trials were compiled in Table 16. The first column of the tables

specifies the particular trial for each gesture, including the optimized or un-optimized trials, or

the baseline case. Next, the total durations in seconds for executing each set of gestures for each

trial are given. The third and fourth column state the initial and final battery levels for each trial,

which are measured by polling the robot’s power state before and after the execution of the

gestures. The fifth column displays the voltage drops from the execution of the gestures, which

are calculated by taking the difference between the initial and final battery levels. The sixth

column includes the depths of discharge calculated from the voltage drops as according to

Equation (12).

The results show that the duration of the optimized plans for the gestures was lower in all cases,

and that the optimizer was successful in choosing motion plans which require less time to

execute. The speed difference when performing the optimized gestures when compared to the un-

optimized gestures ranges from 1% for the point to screen gesture to 20% for the celebration

gesture. The small difference in speed when performing the point to screen gesture is most likely

due to the small range of motion required, making it unlikely that the motion planners would

plan highly variant trajectories for the gesture. On the other hand, the celebration gesture not

only requires large movements from Tangy’s arms, it requires both the right and left arm to

execute the behaviour. This introduces much more possibility for variation in the plans produced

by the motion planners, and a greater difference in execution speeds for an optimized compared

to an un-optimized celebration gesture.

The energy consumption also appeared to be lower among the optimized gestures than the un-

optimized. However, the depth of discharge results seem to indicate that for certain gestures, the

difference in energy consumption between optimized and un-optimized gestures may be small,

such as in the case of the point to screen gesture. For the point to screen gesture, the energy

56

57

consumption from executing the optimal motion plans was indistinguishable from the energy

consumption when executing the un-optimized motion plans. On the other hand, large gestures

such as the celebration gesture can significantly drain the robot’s battery. The difference in depth

of discharge for the optimized compared to the un-optimized celebration gestures was 5% of the

robot’s battery over 25 repetitions of the gesture. Although for some gestures, the optimization

approach offered minimal benefits for providing the most energy efficient plans, the results of

the experiment demonstrate that the optimization technique did provide energy savings across

many repetitions of the same gesture.

57

58

Table 16: Duration and Energy Consumption Differences between Sets of Twenty-Five Executions of Optimized and Un-Optimized Gestures

Point to Screen Gesture

Trial: Total

Duration (s)

Initial Battery Reading (V)

Final Battery Reading (V)

Total Voltage

Drop (V)

Depth of Discharge (%)

Baseline 537 38.5 37.8 0.7 7%

Optimized 537 39.1 38.3 0.8 8%

Un-optimized 542 38.7 37.9 0.8 8%

Celebration Gesture

Trial: Total

Duration (s)

Initial Battery Reading (V)

Final Battery Reading (V)

Total Voltage

Drop (V)

Depth of Discharge (%)

Baseline 890 38.3 37.1 1.2 12%

Optimized 890 38.1 36.2 1.9 19%

Un-optimized 1068 38.9 36.5 2.4 24%

Laugh Gesture

Trial: Total

Duration (s)

Initial Battery Reading (V)

Final Battery Reading (V)

Total Voltage

Drop (V)

Depth of Discharge (%)

Baseline 718 38.8 37.7 1.1 11%

Optimized 718 39.1 37.9 1.2 12%

Un-optimized 797 38.4 37.1 1.3 13%

Wave Gesture

Trial: Total

Duration (s)

Initial Battery Reading (V)

Final Battery Reading (V)

Total Voltage

Drop (V)

Depth of Discharge (%)

Baseline 765 37.9 37 0.9 9%

Optimized 765 39.1 37.9 1.2 12%

Un-optimized 796 38.7 37.3 1.4 14%

The results from the performance review could be used to improve the robot’s behaviour

deliberation system by providing information on the energy consumption of some of its

behaviours. Namely, one potential application of the results from this experiment may be to

allow the robot to factor in the average energy consumption of a single gesture into its

determination of a behaviour. For example, Tangy may decide to perform certain gestures with

relatively higher energy consumption rates fewer times such as the laugh gesture when the

robot’s battery is at a low state of charge.

58

59

6.4 Chapter Summary

Several experiments were conducted to investigate the performance of the robot in a controlled

setting in the research lab. Tangy’s finite state machine performed with 100% success rates in

performing its expected behaviour over all 50 games, provided that the state change was detected

by the robot.

The arm control accuracy was studied by sending Tangy goal arm poses and measuring its

executed pose with an external sensor. The accuracy of the right arm was found to be very high –

Tangy was able to reach within 5% of the targeted goal pose. The left arm appeared to be less

accurate, with one joint reaching within 8% of its targeted orientation. The accuracy of the arms

was most likely impacted by instability in the servos, including vibrations or jerking motions

when performing a movement or attempting to maintain a pose.

The optimization technique in the arm control was tested to investigate its ability to choose the

lowest duration and lowest energy consumption motion plans. Tangy performed several gestures

using the un-optimized motion planner and the optimized motion planner in independent trials.

The results from the trials indicate that the optimization approach was able to choose shorter

duration and lower or equivalent energy consumption plans for the robot.

59

60

Chapter 7 Human Robot Interaction User Studies

A human-robot interaction pilot study was conducted by our research team to investigate the

efficacy and acceptance of Tangy as an autonomous Bingo facilitator for elderly residents in a

long-term care facility. Several Bingo game sessions with the robot took place with several

residents over the course of two weeks. Development of the study’s methodology and analysis of

the results were frequently performed collaboratively with other members of the research team.

This chapter investigates specifically the system performance of the robot, and the reactions from

participants to the Bingo scenario with the robot in this study.

7.1 Participants

Seven residents participated in a total of six Bingo sessions facilitated by Tangy. The residents

were aged 66-79 years old, with a mean of 79.3 years old and standard deviation of 11.7 years.

Participants were chosen to meet the following criteria: they were 1) cognitively intact or with

mild cognitive impairment (Cognitive Performance Scale level of 2 or less [40]), 2) over the age

of 60, 3) fluent in English, and 4) could hear normal levels of speech. Several residents played

multiple sessions with Tangy, with each resident participating in at least two Bingo games.

Participants’ experience levels with computers and robotics were gathered at the end of each

game, with the results listed in Table 17. Most participants had experience with computers, but

only one participant had any experience with robots. Written informed consent was obtained

prior to commencement of the study.

Table 17: Number of Participants with Experience with Computers or Robots

Experience with Computers* Experience with Robots** No experience 1 6

Beginner 2 1 Intermediate 1 0

Advanced 3 0 *Beginner (email, use simple programs) Intermediate (internet, chat) Advanced (editing documents, use complex programs)

** Beginner (seen robots at museums/science centers or stores, or have watched shows with robots) Intermediate (have worked with/used commercial robots) Advanced (have worked on robot developmental aspects including hardware/ software design)

60

61

7.2 Methods

Prior to playing in the Bingo games with Tangy, participants were given a demonstration of the

robot’s capabilities and actions during the game. The demonstration included a game facilitated

by the robot with members of the research team. The purpose of the demonstration was to

introduce the robot and the scenario to the participants so that they would have the opportunity to

present any questions or concerns before playing the game themselves. The demonstration

displayed every potential behaviour that the robot could perform during the game.

The study was conducted in a first floor activity room in the long-term care facility. Each session

lasted approximately 45 minutes to an hour, and consisted of either two or three Bingo games.

The sessions were video recorded for analysis purposes. Tangy’s system performance was

measured using the video recordings of the sessions by other members of the research team. The

system performance review was done in order to assess the execution success rates of the robot

behaviours developed in this thesis in a real-world environment during the study. Two members

of the research team measured the execution success rates by counting the number of times the

robot executed a behaviour which it was supposed to, and when it didn’t execute, or performed

the behaviour incorrectly. This thesis provides a reflection on the execution success rates of the

robot behaviours later on in this chapter.

After each Bingo session, participants were also given a post-interaction questionnaire if it was

their last session. The questionnaire contained closed questions investigating the participants’

acceptance and attitudes towards Tangy. The closed questions were developed by myself and

another member of our group, and categorized into constructs developed from the Almere model

[28] by this member of the research team. These questions were answered using the five-point

Likert scale (where 1= strong disagree, 2= somewhat disagree, 3= neutral, 4= somewhat agree,

5= strongly agree) and are listed in Table 19. The questionnaire also contained open-ended

questions developed in this thesis, which investigated the features on Tangy that residents liked

or found helpful, and other features or activities which they wanted to see on Tangy. Thematic

analysis was performed on the results on the open-ended questions, and the thematic sets are

provided later on in this chapter as well.

61

62

7.3 System Performance Results

The system performance results are documented in Table 18. In general, the success rates for the

robot’s behaviours were 100%, with the exception of two behaviours: the acknowledgement of a

player’s assistance request, and initiating eye contact. Tangy did not acknowledge every request

by nodding at players when two players pressed their buttons at the same time. Upon

investigation, this failure was attributed to the fact that the robot finite state machine did not

include a provision for processing two requests to nod at the same time. If Tangy received one

set of servo commands to nod toward the location of the first assistance request and the second

immediately afterwards, the second set of servo commands would pre-empt the first set.

Tangy failed to perform the eye contact behaviour properly, with instances occurring during the

single user assistance scenarios where the robot would attempt to make eye contact with players

sitting beside the player who requested assistance. The failure of the eye contact was attributed to

the programming logic behind the eye contact module. Tangy would attempt to make eye contact

with the person whose face appears closest to the center of the field of view of the robot.

However, this would only be the correct person to maintain eye contact with if the robot had

navigated perfectly in front of him/her. Should the robot be skewed in orientation or shifted in

location to either the right or the left of the correct player upon initiation of the eye contact

module, it might capture the face of his/her neighbour instead. These failures occurred more

often because participants would frequently move closer to each other to view the robot when it

came to assist a player.

62

63

Table 18: System Performance Results1

Robot State Robot Location Card State

Expected Robot Behaviour

Success Rate

Pre-Game At front of room - Greet players 100% Multi-User Interactions At front of room - Call out Bingo

number 100%

Multi-User Interactions At front of room - Provide Jokes and

Facts 100%

Multi-User Interactions At front of room Acknowledge

player(s) request(s) 92.68%

Transition At front of

room/At another player’s location

- Navigate towards location of player 100%

Single User Interactions

At player’s location -

Localize player’s face and initiate eye

contact 63.64%

Single User Interactions

At player’s location

Occluded card

Prompt player to move card 100%

Single User Interactions At player’s

location Incorrectly

marked

Request to remove marker(s) from

incorrectly marked numbers

100%

Single User Interactions At player’s

location Missing markers

Request to mark missing number(s) on

the Bingo card 100%

Single User Interactions At player’s

location Correctly marked

Provide encouragement 100%

Single User Interactions

At player’s location

Winning card

Provide congratulations 100%

Post-Game - - Say farewell to player 100% 1Measured by two other members of the research team

7.4 Human-Robot Interaction Results

7.4.1 Participant Questionnaire Results

The statements measured with the Likert scale for the questionnaire are presented in Table 19

with the mean scores for each question.

63

64

Table 19: Post-Bingo Session Questionnaire Results2

Statement Mean I enjoy playing Bingo with Tangy. 5.00

I think Tangy could help me during the game. 4.00 Tangy is able to help me. 4.57

I will play Bingo with Tangy again. 5.00 I will ask Tangy for help again. 4.71

I will ask Tangy for help in the future. 5.00 I think Tangy should host Bingo games again. 4.86

I think Tangy is helpful to other players. 4.86 I think Tangy makes the Bingo game interesting 4.86

I like Tangy’s appearance 4.71 It feels like Tangy is looking at me when I am

playing the game with it 4.00

I am comfortable interacting with Tangy 5.00 I find Tangy intimidating* 1.00

I trust Tangy’s help 4.86 I will follow what Tangy asks me to do in the

Bingo game 5.00 2Questionnaire developed collaboratively with another member of the research team; statistical analysis done by same member

The responses to the open-ended questions from the questionnaire revealed that participants

wished to see Tangy facilitate recreational games which were more popular among residents,

such as card games, board games, or physical activities. Tangy’s voice was praised by three

participants, who indicated that it was “clear” and “human-like”. Tangy’s arm gestures were

singled out by two residents as very enjoyable; one participant stated that the robot’s body

language was very expressive and the robot’s celebration dance demonstrated clearly that

“Tangy was excited”. Several participants noticed Tangy looking at them when they were

playing the Bingo game; one participant described “[I] really enjoyed when Tangy looked at

[me] during the game”. An aspect of the game considered to have a negative impact on four

participants’ enjoyment of the session was the slow pacing of the game by the robot facilitator. A

participant gave his opinion: “Personally, for me, I would like the game to be faster. But I think

it was a good speed for the others.” Finally, the majority of participants indicated that they had

no preference for either the verbal communication of the robot or the visual display of

information on its tablet, but that both were important sources of information during the game.

64

65

7.5 Discussion

The robot was demonstrated to be consistent in properly executing all of its behaviours at the

appropriate times during the Bingo game except the acknowledgement behaviour and the eye

contact behaviour. The failure of the acknowledgement behaviour can be avoided in the future

by adding a provision for multiple consecutive nods in the code when sending servo commands

to the robot’s neck. The provision should preserve the intent of the acknowledgement behaviour,

which is to signal to every player who requested assistance that the robot has detected his/her

request; as such, the provision should simply queue up nodding gestures and complete them one

after the other. The failure of the eye contact behaviour requires a more complex solution. As

non-zero tolerances must be assumed with the robot’s navigation system, Tangy cannot be

assumed to always position itself with the face of the player who requested assistance directly in

the center of its field of view. One possible method to prevent this failure may be to store

player’s faces within the robot’s database and link them uniquely to the assistance request

devices. Thus, the robot would be able to immediately know who requested assistance even

before navigating to the player. It would then be able to choose the correct player within its field

of view to focus on and initiate eye contact with.

The results of the post-game questionnaire showed that the participants enjoyed the Bingo

session with Tangy as the facilitator and would be willing to play Bingo with the robot again in

the future. The comfort participants felt when interacting the robot may have influenced their

enjoyment of these sessions and their desire to interact with the robot in the future [48]. As

comfort has been linked with the sociability of a robotic system, the sociability of Tangy may

have allowed participants to feel more comfortable interacting with the robot [50]. In [50], the

expressive robot iCat had two types of conversations with older adults at a long-term care

facility: one type which involved social behaviours such as gazing at users and using facial

expressions, and the other which didn’t involve the social behaviours. The study demonstrated

that older adults felt more comfortable with the more sociable robot. Similarly, in our study,

participants indicated that they were comfortable and not intimidated by interacting with Tangy.

This may have been motivated by the various types of social behaviours performed by the robot

including speech, eye gaze and physical gestures, which participants alluded to consistently in

their responses to the open ended questions in the post-game questionnaire.

65

66

Participants’ feedback on the Bingo session also provided insights into potential improvements

on Tangy’s features. Several comments about the robot’s pacing of the game and of the slow

speed of its interactions with single users came up during the post-game questionnaire. A

possibility for increasing the robot’s pacing when facilitating the Bingo game can be to increase

the speed at which Tangy calls out Bingo numbers. The single user interactions can also be sped

up by increasing the navigation speed of the robot through the environment. All participants

indicated that they enjoyed the supplementary behaviours such as the robot’s jokes and

background music. Their enjoyment of these features can inform Tangy’s design by increasing

the amount of these engaging interactions in the design of the robot’s behaviours.

7.6 Chapter Summary

This chapter presented the human-robot interaction pilot study done at a long-term care facility

to investigate the efficacy and acceptance of Tangy as a robot facilitator of the recreational

activity Bingo among long-term care residents. Tangy facilitated six Bingo sessions with 7

residents. The sessions were analyzed for the robot’s system performance, participant

compliance with Tangy’s requests, and participant engagement with the activity. The results

demonstrated that Tangy was able to perform most of its behaviours appropriately during the

activity and participants indicated that they enjoyed playing the game with Tangy. Several

participants explained that they especially enjoyed some of Tangy’s social features. They

brought up the clearness of the robot’s voice, the usefulness and expressiveness of the robot’s

gestures, and their enjoyment of the eye contact during the game. Participants mentioned that the

speed of the Bingo game and of the robot’s interactions with players could be increased. Overall,

participants indicated that they would interact with Tangy again in the future, which may be due

to their high level of comfort with the robot.

66

67

Chapter 8 Conclusion

8.1 Summary of Contributions

This chapter provides a summary of the developmental work on Tangy’s behaviours and

actuation capabilities and the experiments designed and implemented to investigate their

performance and efficacy for older adults.

8.1.1 Gathering End-User Feedback

Focus group studies were performed at a long-term care facility and a retirement home in order

to obtain the opinions and design considerations of elderly residents, family members and

healthcare staff about socially assistive robots and Tangy. Participants were shown a demo of an

early implementation of the robot, and asked to give feedback on the robot’s activities, its

features and the physical robot itself. The suggestions participants gave about the various

activities the robot could do, including recreational activities other than Bingo, prompting and

reminding, simple interactions with residents with cognitive impairments, and physical aid

around the facilities, provides a large amount of potential future work for implementing socially

assistive robots in long-term care. Moreover, important feedback was gained about the

interaction methods between the robot and residents. The results of the study indicated that

participants strongly believed the robot’s interface needed to be highly intuitive and easy to use.

Participants also brought up the issue of accessibility for physically impaired residents, which

needed to be considered when designing the interaction methods between residents and the robot.

8.1.2 Assistive Robotic Behaviours and Actuation Capabilities

The behaviours and the behaviour determination approach for the facilitation of the recreational

activity Bingo game were developed and implemented on Tangy. The behaviours of the robot

were chosen to imitate human Bingo facilitators at a long-term care facility and a retirement

home. The assistive social robot behaviours uniquely covered both the crucial multi-user and

single user interactions that a facilitator may have with players during the facilitation of a Bingo

game in order to allow the robot to facilitate the game completely autonomously. The robot

performs five major types of behaviours depending on the current junction of the game: i) pre-

67

68

game introduction; ii) multi-user behaviours; iii) transition actions; iv) single user assistive

behaviours; and v) post-game farewell. Which behaviours were chosen to be executed was

determined by the robot’s finite state machine. Tangy’s FSM determined appropriate behaviours

based on the state changes in the world, robot or detected card of a player who requested

assistance.

The other contribution made in this thesis relates to the actuation modules which allow Tangy to

perform the assistive behaviours of verbally communicating with players, making eye contact

with players when assisting them, and physically gesturing. The robot uses a synthesized voice in

order to call out Bingo numbers, instruct players on fixing their misplayed Bingo cards, praise

and encourage correctly played cards, and congratulate players with winning cards. Tangy

maintains eye contact by tracking a player’s face in the 2D video stream from the webcam in its

eye and then reorienting its head in order to keep that player’s face as close to its direct line of

sight as possible. Finally Tangy gestures using its arms by executing trajectory plans created by

motion planners. Tangy chooses the optimal motion plan with the lowest duration or the lowest

energy consumption requirements.

8.1.3 Experimental Results

Tangy’s system performance with respect to its finite state machine was evaluated by having the

robot run the equivalent of 50 Bingo games and performing every behaviour possible in the

experiment. Tangy’s arm control was assessed in both the accuracy of each individual joint and

the effectiveness of the optimization technique used to select the lowest duration and energy cost

motion plans. The experiments demonstrated that the arm servos were fairly accurate, with each

joint the right arm reaching within 5% of a pre-determined goal movement and each joint in the

left arm reaching within 8% of the same goal movement. Moreover, the experiments

demonstrated that the optimization approach to the arm control succeeded in choosing the plans

with the shortest duration and the lowest battery voltage consumption.

8.2 Discussion of Future Work

Future work for the implementation of Tangy should include investigating the possibility of the

robot facilitating other recreational activities other than Bingo. This thesis develops several

Bingo-specific behaviours, but also social and assistive behaviours which could be implemented

68

69

in other activities, like the popular trivia and trivia-like activities in the long-term care and

retirement home facilities.

Future research should also investigate the efficacy of the social robot behaviours for elderly

residents during the Bingo game in order to inform the design of social behaviours for

autonomous socially assistive robots for the elderly.

Finally, future work can be done on creating an integrated system which can use the energy

consumption values found in the experiment described in section 6.3. The integrated system can

not only choose the optimal motion plan for the execution of an arm gesture, but also optimize

the usage of gestures in order to increase the runtime of the robot on a single charge of its

batteries.

69

70

References

[1] I. Deary et al, “Age-associated cognitive decline”, British Medical Bulletin, Vol. 92, No.1,

pp. 135-152, 2009.

[2] G. Lafortune and G. Balestat, “Trends in severe disability among elderly people: assessing

the evidence in 12 OECD countries and the future implications,” OECD Publishing, 2007.

[3] S. Sharkey, “People Caring for People: Impacting the Quality of Life and Care of

Residents of Long-Term Care Homes,” Ontario Ministry of Health and Long-term Care,

Toronto, ON, 2008.

[4] B. Sobel, “Bingo vs. physical intervention in stimulating short-term cognition in

Alzheimer’s disease patients”, American Journal of Alzheimer’s Disease and Other

Dementias, Vol. 16, No. 2, pp. 115-120, 2001.

[5] F. Colombo et al., “Help wanted? Providing and paying for long-term care,” OECD

Publishing, Paris, 2011.

[6] United Nations, “World Population Prospects The 2015 Revision,” Key Findings and

Advance Tables, New York, 2015.

[7] E. Farina, et al, “Evaluating two group programmes of cognitive training in mild-to-

moderate AD: Is there any difference between a ‘global’ stimulation and a ‘cognitive-

specific’ one?”, Aging and Mental Health, Vol. 10, No. 3, pp. 211-218, 2006.

[8] K. Ball, D. Berch, and K. Helmers, "Effects of Cognitive Training Interventions with Older

Adults", The Journal of the American Medical Association, Vol. 288, No. 18, pp. 2271-

2281, 2002.

[9] R. Anand et al., “Brain health fitness: beyond retirement”, Educational Gerontology, Vol.

37, No. 6, pp. 450-465, 2011.

[10] N. Gates and M. Valenzuela, “Cognitive exercise and its role in cognitive function in older

adults”, Current Psychiatry Reports, Vol. 12, No. 1, pp. 20-27, 2010.

[11] G. Smith et al., “A Cognitive Training Program Based on Principles of Brain Plasticity:

Results from the Improvement in Memory with Plasticity‐based Adaptive Cognitive

Training (IMPACT) Study”, Journal of the American Geriatrics Society, Vol. 57, No. 4,

pp. 594-603, 2009.

70

71

[12] A. Kueider et al., “Computerized cognitive training with older adults: a systematic

review”, PLoS One, Vol. 7, No. 7, pp 1-13, 2012.

[13] A. Spector, et al., “Efficacy of an evidence based cognitive stimulation therapy programme

for people with dementia”, British Journal of Psychiatry, Vol. 183, pp. 248–254, 2003.

[14] V. Breuil, et al., “Cognitive stimulation of patients with dementia: Preliminary results.

International Journal of Geriatric Psychiatry”, Vol. 9, pp. 211–217, 1994.

[15] O. Zanetti, et al., “Effectiveness of procedural memory stimulation in mild Alzheimer’s

disease patients: A controlled study”, Neuropsychological Rehabilitation, Vol. 11, pp. 263-

272, 2001.

[16] S. Zarit, J. Zarit, & K. Reever, “Memory training for severe memory loss: Effects on senile

dementia patients and their families” Gerontologist, Vol. 22, pp. 373–377, 1982.

[17] V. Tesky, C. Thiel, W. Banzer, and J. Pantel, “Effects of a group program to increase

cognitive performance through cognitively stimulating leisure activities in healthy older

subjects,” GeroPsych: The Journal of Gerontopsychology and Geriatric Psychiatry, Vol.

24, No. 2, pp. 83-92, 2011.

[18] V. Menec, “The relation between everyday activities and successful aging: A 6-year

longitudinal study”, The Journals of Gerontology Series B: Psychological Sciences and

Social Sciences, Vol. 58, No. 2, pp. S74-S82, 2003.

[19] T. Laudate. Neargarder et al., “Bingo! Externally supported performance intervention for

deficient visual search in normal aging, Parkinson's disease, and Alzheimer's disease,”

Aging, Neuropsychology, and Cognition, Vol. 19, No. 1-2, 102-121, 2012.

[20] T. Glass et al., “Social engagement and depressive symptoms in late life longitudinal

findings”, Journal of Aging and Health, Vol. 18, No. 4, pp. 604-628, 2006.

[21] J. Unger, C. Johnson, and G. Marks, “Functional decline in the elderly: evidence for direct

and stress-buffering protective effects of social interactions and physical activity”, Annals

of Behavioural Medicine, Vol. 19, No. 2, pp. 152-160, 1997.

[22] D. McColl, W. G. Louie, and G. Nejat, “Brian 2.1: A Socially Assistive Robot for the

Elderly and Cognitively Impaired,” IEEE Robotics & Automation Mag., Vol. 20, No. 1, pp.

74-83, 2013.

71

72

[23] A. Tapus, C. Tapus, and M. Mataric, “The use of socially assistive robots in the design of

intelligent cognitive therapies for people with dementia,” IEEE Conference on

Rehabilitation Robotics, pp. 924-929, 2009.

[24] Y. Oida, et al., “Development of a Robot-Assisted Activity Program for Elderly People

Incorporating Reading Aloud and Arithmetic Calculation,” Asian Perspectives and

Evidence on Health Promotion and Education, pp. 67-77, 2011

[25] M. Kanoh et al., “Examination of practicability of communication robot-assisted activity

program for elderly people”, Journal of Robotics and Mechatronics, Vol. 23, No. 1, pp. 3-

12, 2011.

[26] T. Hamada, et al., “Robot therapy as for recreation for elderly people with dementia-Game

recreation using a pet-type robot”, in IEEE International Symposium on Robot and Human

Interactive Communication, pp. 174-179, 2008.

[27] R. Khosla et al., “Embodying care in Matilda: an affective communication robot for the

elderly in Australia,” in ACM SIGHIT Proceedings of International Health Informatics

Symposium, pp. 295-304, 2012.

[28] M. Heerink, et al., "Assessing acceptance of assistive social agent technology by older

adults: the almere model", International Journal of Social Robotics, Vol. 2, No. 4, pp. 361-

375, 2010.

[29] B. De Ruyter, et al. "Assessing the effects of building social intelligence in a robotic

interface for the home." Interacting with computers, Vol. 17, No. 5, pp. 522-541, 2005.

[30] D. Miyauchi, et al. "Human-robot eye contact through observations and actions." Pattern

Recognition, ICPR 2004. Proceedings of the 17th International Conference on. Vol. 4,

2004.

[31] A. Kendon, “Some functions of gaze-direction in social interaction”, Acta Psychologica,

26:22–63, 1967.

[32] T. Yonezawa et al., “Gaze-communicative Behaviour of Stuffed-toy Robot with Joint

Attention and Eye Contact based on Ambient Gaze-tracking”, Proceedings of the 9th

international conference on Multimodal interfaces, pp. 140-145

[33] H. Ishiguro et al., “Development of an Interactive Humanoid Robot “Robovie” – An

interdisciplinary approach”, Robotics Research, Springer Berlin Heidelberg, pp.179-191,

2003.

72

73

[34] C. Breazeal, et al. “Learning from and about others: Towards using imitation to bootstrap

the social understanding of others by robots”, Autonomous Robots, Vol. 11, No. 1-2, 2005

[35] C. Sidner, et al. “Explorations in engagement for humans and robots.” Artificial

Intelligence, Vol. 166, No. 1, pp. 140-164, 2005.

[36] C. Breazeal, et al. "Effects of nonverbal communication on efficiency and robustness in

human-robot teamwork." Intelligent Robots and Systems, IEEE, 2005.

[37] W. Bainbridge, et al. "The effect of presence on human-robot interaction." Robot and

Human Interactive Communication, 2008. RO-MAN 2008. The 17th IEEE International

Symposium on. IEEE, 2008.

[38] H. Miwa, et al., “Effective Emotional Expressions with Emotion Expression Humanoid

Robot WE-4RII,” in Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings.

2004 IEEE/RSJ International Conference on, Vol.3, pp. 2203-2208, 2004.

[39] M. Haring, N. Bee and E. Andre, "Creation and evaluation of emotion expression with

body movement, sound, and eye color for humanoid robots," in IEEE International

Symposium on Robot and Human Interactive Communication, pp. 204-209, 2011.

[40] G. Louie et al., “A Focus Group Study on the Design Considerations and Impressions of a

Socially Assistive Robot for Long-Term Care”, Robot and Human Interactive

Communication, 2014 RO-MAN: The 23rd IEEE International Symposium, 2014.

[41] J.N. Morris et al., “MDS cognitive performance scale,” Journal of Gerontology, Vol. 49,

No. 4, pp. M174-M182, 1994.

[42] I. Sucan, and S. Chitta, “MoveIt!”, [Online] Available: http://moveit.ros.org [Accessed

March 2015], 2014.

[43] I. Sucan, M. Moll, and L.E. Kavraki, “The Open Motion Planning Library”, IEEE Robotics

& Automation. Mag, Vol. 19, No. 4, pp. 72–82, 2012.

[44] R. Murray, et al. A mathematical introduction to robotic manipulation. CRC press, 1994.

[45] ServoCity, Available: https://www.servocity.com/html. [Accessed Aug. 2015].

[46] Omron, “OKAO Vision”, Available: http://www.omron.com/r_d/coretech/vision/okao.html

[Accessed May 2013], 2007.

[47] Simple DirectMedia Layer, [Online] Available: https://www.libsdl.org [Accessed March

2015]

73

74

[48] Dr. Robot Inc., [Online] Available: http://www.drrobot.com/products_H20.asp [Accessed Sept. 2015]

[49] E. Broadbent, E., R. Stafford, and B. MacDonald, “Acceptance of healthcare robots for the

older population: Review and future directions,” International Journal of Social Robotics,

Vol. 1, No. 4, pp. 319-330, 2009.

[50] M. Heerink et al., “The influence of a robot's social abilities on acceptance by elderly

users,” IEEE International Symposium on Robot and Human Interactive Communication,

pp. 521-526, 2006.

74