2016 The 2nd International Conference on Control, Automation and Robotics

Cloud-assisted Humanoid Robotics for Affective Interaction

Yujun Ma l ,2, Yixue Rao, Yongfeng Qian, and Min Chen ISchool of Computer Science and Technology

Huazhong University of Science and Technology Wuhan, China

2Computer Network Center Nanyang Institute of Technology

Nanyang, China e-mail: {yujun.hust. yixue.epic. yongfeng.epic}@gmail.com. minchen@ieee.org

Abstract-In recent years, the humanoid robot is received great attention, and gradually develop to households and personal service field. The prominent progress of cloud computing, big data, and machine learning fields provides a strong support for the research of the robot. With affective interaction ability of robot has a broad market space and research value. In this paper, we propose a cloud-assisted humanoid robotics for affective interaction system architecture, and introduce the essential composition, design and implementation of related components. Finally, through an actual robot emotional interaction test platform, validating the feasibility and extendibility of proposed architecture.

Keywords-cloud computing; humanoid robotics; affective interaction.

I. INTRODUCTTON

In recent years, robot research has become one of the most popular research fields in industry and academia. Especially the humanoid intelligent robot, it has drawn a great concern. While promoting industrial robots, countries all around the world are paying more and more attention to the intelligent service robot [1]. The application domain of robots is developing gradually from industry to family and personal service. With the development of research on robot products with autonomous ability and intelligence, the human will be free from the simple, monotonous, and dangerous labor. At the same time, it will also bring a new round of technological innovation boom, and become an important economic growth point.

Humanoid robot has made great progress, but it is also facing many technical challenges to make the robot fully integrated into human life, among which, to equip the human robot with emotional interaction ability is one of the most challenging problems [2]. In the past, most of the robots are to complete repetitive work in accordance with the programs written in advance. They usually do not have the autonomous capability. They are not intelligent or their intelligent level is very low. They cannot aware of human feelings. Therefore, they lack initiative interaction with people, and cannot make initiative emotional interaction with people based on their emotions. It greatly restricts the promotion of robot applied to the field of horne and service.

In recent years, with the maturity and application of c10ud computing and big data technology, as weil as the breakthrough in machine learning research, the level of artificial intelligence has been greatly improved, which has provided a powerful technical support for the research of intelligent robot with the ability of emotional interaction.

Cloud computing is a new type computing and service mode based on Internet [3] , [4]. Through this method, pooling hardware and software resources and information can be supplied to the service requestor on a basis of requirement. The traditional robots are always restricted in the hardware and software functions where there are serious problems. But the c10ud computing, as a good support to the robot technology can easily combine the cloud computing with the robot technology to build a c10ud robot [5], [6] . As a front end equipment, the robot takes charge in signal collection, specific action performance and some simple tasks of analysis and process while the more complicated tasks which need a large scale computing cluster will depend on the c1oud. The c10ud itself has strong storage and calculating ability, training, learning and building effective models by advanced machine learning algorithm and transmitting the results of calculation or analysis back to the robots. In that way, the robot will be provided with the second wise brain with the help of c1oud' s strong analysis and processing abilities.

In this paper, we study the robot with the ability of emotion interaction by combining the c10ud computing technology with the humanoid robot. The new technology of intelligent perception and cognition is integrated into the robot research to enhance the robot intelligence level. The paper has proposed an emotional interactive robot system architecture based on c10ud computing technology, constructed the prototype system of an emotional interactive robot taking humanoid robot as the service front end, and taking the back end c10ud platform as the robot's super brain, verified the feasibility of the proposed architecture, and evaluated the operating results and user experience of the system.

The rest of this paper is organized as folIows. Section 11 presents related works. The c1oud-assisted humanoid robotics for affective interaction system architecture is presented in Section III. Section IV describes the design and

978-1-4673-9859-6116/$31.00 ©2016 IEEE 15

implementation of the front-end robot and the software system. Section V introduces the evaluation platform of the cloud auxiliary affective interaction robot. Finally, Section VI concludes this paper.

n. RELATED WORK

Humanoid robot has been studied for a long time. In recent years, there are some related research achievements and products in the field of elderly nursing, medical treatment, entertainment and industry. KASPAR [7] humanoid robot has a similar appearance to the child and focuses on the human computer interaction. It is used in the treatment of children with autism. Alemi and his team have applied Humanoid robot to the treatment of pediatric cancer [8]. The second generation of Robonaut robot developed by General and Motors NASA is the fust humanoid robot used in space [9]. It is used to assist astronauts in the maintenance and maintenance tasks of the international space station. Kofinas and his team [10] have launched a Kinematics research of robot based on Aldebaran NAO humanoid robot [11 ].

To combine the c10ud computing with strong functions with robot research is one of the hot researches in recent years. Yongquan and his team had designed a Robot Cloud Center (RCC) framework [12], adopting the general c10ud computing model to break the computing processing capacity and multi functions limitation of the robot. In the RCC framework, robots are treated as a kind of public service, and everyone can access powerful robot in a relaxing, efficient and inexpensive way. In the literature [13], the author has proposed a framework based on information fusion. Through the network, many robots are connected to form a networked robot system to accomplish more computing intensive tasks. The design uses c10ud to provide users with powerful processing capabilities, while ensuring the simplicity of distributed robots. Kehoe and his team have shown a c10ud robot using Google's object recognition algorithm [14]. In the literature [15], the author has proposed a distributed agent framework with group intelligence. A variety of heterogeneous robots can work together. Robot operating system (ROS) [16] is used to collect sensing data and communication. The author uses Hadoop as the c10ud computing and storage platform, and had implemented a grid based FastSLAM algorithm based on the Map/Reduce computing model ofHadoop.

The ability of emotional interaction is an important function of service robot, which has great market potential and research value. Wafa and his team have introduced a cognitive and emotional interaction oriented social human machine interaction framework CAIO, and used the robot to analyze the interlocutors mental (including emotion) state [17]. Min et al. have proposed AIWAC framework [18], a kind of emotional interaction system based on wearable computing and c10ud computing. In the literature [19], the author has studied the emotional interaction between human and humanoid robot when they shake hands. This is a kind of human machine emotion interaction mode based on touch and it has integrated the robot's facial expression and emotional expression.

16

In summary, although there are some researches about to enhance the service capabilities of the robot through the c10ud computing and some studies on humanoid robot with emotional interaction ability, but these studies are either to use c10ud computing to improve a single ability of a robot (e.g., tracking and identification of the target), or to use c10ud computing to achieve a networked distributed robot system. However, there are few researches focusing on the c10ud assisted emotional interaction humanoid robot. Next, we will introduce a general humanoid emotional interaction robot system framework based on c10ud computing technology, and then elaborate the various components ofthe framework in detail. Finally, we will introduce a prototype test system based on this framework, which is developed by uso

III. HUMANOID ROBOTICS FOR AFFECTIVE INTERACTTON

SYSTEM ARCHTTECTURE

In order to make the humanoid robot have emotional interaction, we need to analyze a large amount of data related with emotions (such as human's expressions, languages, actions, words and pictures used in social network.), but the robot is unable to complete the task of resource analysis due to its limited processing and storage capacity. Cloud computing has changed the traditional software delivery model. It provides computing, storage and network resources to the user in a way of service. In this paper, we have proposed a humanoid emotional interaction robot system framework based on cloud computing, in which the resources consuming storage and analysis of large scale emotional data will be unloaded to the back-end cloud platform. The front end humanoid robot is only responsible for simple tasks, such as data acquisition, preprocessing and uploading to the c1oud. Robot is the carrier of emotion interaction, which is responsible for receiving interactive instruction from the c1oud, and realizing emotional interaction with the user through its own software and hardware.

The system architecture is divided into three layers, i.e. user, robot and c10ud service. User means the user of the robot, and also the object of the emotional interaction. In order to improve the accuracy of emotion recognition, the user will be required to wear wearable devices (such as smart watch, or smart bracelet) to collect user's physiological indicators (e.g., body temperature, heart rate and gait, etc.). The data acquired by these wearable devices is sent to the back-end cloud platform through the robot. As a large number of computing and storage tasks are completed by the cloud platform, the reliability of the network connection between the robot and the cloud platform is particularly important. Therefore, in order to ensure the smooth communication ofthe robot and the cloud platform, the robot needs to be inserted with a variety of wireless communication modules (4G-LTE [20], 3G, WiFi), and can automatically switch network connection in different communication modules. In order to realize the emotional interaction, the robot needs to be configured with audio and video sensors, LED lights and other components. The robot body must be flexible

enough to complete a variety of actions. In order to perceive the environment information of the user's position, the robot also needs to be integrated with a variety of environmental sensors. Cloud platform adopts big data storage and computing engine based on Hadoop, with the help of sentiment analysis algorithm based on deep learning and integrates the data collected by the robot and the user's social network data to complete the analysis and the predication of the user's emotion condition. The cloud issues the emotion analysis results and the emotional interaction instructions to the robot, and makes the robot to complete the specific interactive tasks. During the emotional interaction, the robot will report the effect of real time emotional interaction to the platform. The cloud platform adjusts the emotional interaction timely according to the robot's feedback.

IV. SYSTEM DESIGN AND IMPLEMENTATTON

Emotional interaction of humanoid robot is a very complicated system. Following we will introduce the design and implementation of the whole system from three aspects: the hardware structure of the robot, the related software system and the back-end cloud platform.

A. Hardware Architecture of Humanoid Robotics

The hardware components of the real robot we designed include head, upper limb, leg, sensing device, communication, etc.

B. Cloud System of Humanoid Robotics

- . ;"liill~

n

---+- Data from thc robot: Facial E;"prcssion, Video, Voicc, User action, Hcm1 rate, Ell\'irolUncnul data eie

~ Data from tb: weil! nctwork: Tc:>..i., lunge, Video, Social rcblDil'l

Figure I. Cioud system architecture of humanoid robotics for affective interaction.

The cloud system is the core of the humanoid robot intelligence and the storage center of data. As shown in Fig. 1, the cloud receives a steady stream of data from robots and

: .. .......... ... .... .... ....... :

social networks. Data from the robot includes the robot's perception of the surrounding environment, human physiological signals, and other related data (such as human expressions, sounds, movements, etc.). Social network data sources include the user's Micro blog, shared photos, videos, etc. Because there are a wide variety of data, the data needs to be distributed to the corresponding storage engine (including relational database and HDFS) according to the types of the data and the requirements of sentiment analysis. This part of work is done by the Data Dispatcher module. Relational database is mainly to preserve the date that is structured, required to be queried quickly, and is not very I arge. HDFS is to preserve the date that is non-structured and is very large (for example, the related audio and video). In addition to the long-time preserved data, there are some data that can be directly used for emotion recognition. After the matching operation of the emotion recognition model based on support vector machine (SVM), the recognition results are sent to the emotional interaction module. Emotion matching model is obtained by the large data processing, analysis engine and the machine learning algorithm which supports emotion recognition.

C. Humanoid Robotics Software System

We have developed relevant robot control software based on jJ C/OSII and application software based on Android platform. Local apps operate on LTE Mobile Phone of robot, and users' LTE-enabled smart mobile terminaloperates remote management software of robot, to establish network communication with robot by L TE network of telecom operator. And cloud platform operates apps related to robot intelligence. The mainly software modules are folIows: humanoid robotics control, VI module, voice recognition module, sensor data processing and display module and remote video transmission module.

V. EVALUATION PLATFORM OF CLOUD AUXTLTARY

AFFECTIVE INTERACTTON ROBOT

This paper has put forward the realization of strong treatment capability of cloud based on the robot architecture we have raised, and we achieved a demonstration application of L TE-controlled emotion interaction robot as Fig. 2 shows based on this architecture.

Figure 2. Robot affection interaction based on c10ud computing.

17

The robot captures video by camera in real time, and positions out human facial expressions by captured videos, and then sends the expression images to remote c10ud platform for expression identification. The cloud platform can train the human facial expression module by large amount of human facial data, and recognize the facial expressions transmitted by robot based on trained module so as to identify the emotional states of human-being (such as: happy or depressed). At the same time, robot also accepts the user' s voice and physiological signals (e.g., heart rate, blood press ure ), and the surrounding environment signal, and then sends the data to the c1oud. The cloud also collects user data in various kinds of social networks. Finally, the cloud predicts the user's emotional state by data fusion analysis based on user's data.

The cloud platform sends affective interaction instructions to front-end robot based on the results of emotional prediction so as to let the robot has targeted interaction with users, as shown in Fig. 2.

In order to evaluate the robot can affect people's emotion and let the users feel better. 10 volunteers (including 4 males and 6 females) join the experiment, their ages from 23 to 30 and average age is 25.2. We evaluated our system by two ways. First, we recorded the volunteers ' facial expression with video camera and analyzed it. Second, each volunteer evaluated robot actions, play comedy movies, play upbeat music, robot action and play comedy, robot action and play upbeat music for the emotion care. In Fig. 3(a) shows 5 kinds ofways to care user's emotion. The recognition ofhappiness according to face recognition. This result shows that robot action and play comedy movies played a good effect of emotion care for volunteer. In Fig. 3(b) shows the volunteers evaluated 5 kinds of ways emotion care and give the score range 0 to 100. As can be found from the Fig. 3(a) and Fig. 3(b), the system performance and the evaluation of volunteers are very similar.

(a) Volunteers' reaction (b) Volunteers'

Figure 3. Case study about the robot in emotion care.

The robot with emotional care function can be used in empty-nesters care, patient recovery, safety monitoring, adjunctive therapy and safe driving; besides, in professions with great work pressure, it can help to relieve staff's pressure so as to improve work efficiency.

VI. CONCLUSION

A cloud-assisted humanoid robotics architecture for affective interaction based on L TE is put forward in this

18

paper. We introduce all components of robot in details, designs and realizes the robot system according to the proposed architecture. Furthermore, an emotion interaction robot demonstration system is achieved. Finally, validating the feasibility and extendibility of robot architecture through the actual robot emotional interaction test platform.

ACKNOWLEDGMENTS

This work was supported by Henan province science and technology research projects No. 132102210483.

REFERENCES

[1] J. Cheng, X. Chen, and P. Lukowicz, 'Towards Coexistence of Human and Robot: How Ubiquitous Computing Can contribute?" in Robot Intelligence Technology and Applications 3. Springer, 2015, pp. 425-431

[2] B. Chandrasekaran and J. M. Conrad, "Human-robot collaboration: A survey," in SoutheastCon 2015. IEEE, 2015, pp. 1-8

[3] L. Zhou and H. Wang, "Toward Blind Scheduling in Mobile Media Cloud: Fairness, Simplicity, and Asymptotic Optimality," IEEE Transactions on Multimedia, vol. 15, no. 4, pp.735-746, June 2013.

[4] L. Zhou, Z. Yang, J. Rodrigues, and M. Guizani, "Exploring Blind Online Scheduling for Mobile Cloud Multimedia Services," IEEE Wireless Communications, vol. 20, no. 3, pp. 54-61 , June 2013 .

[5] C. Lai, H. Chao, Y. Lai, J. Wan, "Cloud-assisted Real-time Transrating for HTTP Live Streaming", IEEE Wireless Communications, Vol. 20, No. 3, pp. 62-70, June 2013.

[6] C. Lai, H. Wang, H. Chao, G. Nan, "A Network and Device Aware QoS Approach for Cloud-Based Mobile Streaming", IEEE Transactions on Multimedia, Vol. 15, No. 4, pp. 747-757, June 2013.

[7] S. Costa, H. Lehmann, K. Dautenhahn, B. Robins, and F. Soares, "Using a humanoid robot to elicit body awareness and appropriate physical interaction in children with autism," International Journal of Social Robotics, vol. 7, no. 2, pp. 265-278, 2015.

[8] M. Alerni, A. Ghanbarzadeh, A. Meghdari , and L. J. Moghadam, "Clinical Application of a Humanoid Robot in Pediatric Cancer Interventions," International Journal ofSocial Robotics, 2015.

[9] M. A. Diftler, J. Mehling, M. E. Abdallah, N. A. Radford, et al. , "Robonaut 2-the first humanoid robot in space," IEEE ICRA, 2011 pp. 2178-2183.

[10] N. Kofinas, E. Orfanoudakis, and M. G. Lagoudakis, "Complete analytical forward and inverse kinematics for the NAO humanoid robot," Journal of Intelligent & Robotic Systems, vol. 77, no. 2, pp. 251-264, 2015.

[ll] D. Gouaillier, V. Hugel, P. Blazevic, et al. , "Mechatronic design of NAO humanoid," in Robotics and Automation, 2009. ICRA '09. IEEE International Conference on. IEEE, 2009, pp. 769-774.

[12] Y. Liang, Z. Zhang, J. Fan, et al , "A Construction Strategy for Cloud Robotic System," Sensor Letters, vol. 12, pp. 255-259(5), 2014.

[13] B. Liu, Y. Chen, E. Blasch, et al, "A holistic c1oud-enabled robotics system for real-time video tracking application," in Future Information Technology. Springer, 2014, pp. 455-468.

[14] B. R. Kehoe, "Cloud-based Methods and Architectures for Robot Grasping," Ph.D. dissertation, UNIVERSLTY OF CALIFORNIA, BERKELEY, 2014.

[15] R. Arumugam, V. R. Enti , L. Bingbing, et al. , "DAvinCi: A c10ud computing framework for service robots," IEEE ICRA, 2010, pp. 3084-3089.

[16] M. Quigley, K. Conley, B. P. Gerkey, et al. , "ROS: an open-source Robot Operating System," In Open-Source Software workshop ofthe International Conference on Robotics and Automation (ICRA), 2009.

[17] W. Johal, D. Pellier, C. Adam, et al. , "A Cognitive and Affective Architecture for Social Human-Robot Interaction," in Proceedings of

the Tenth Annual ACM/IEEE International Conference on Human­Robot Interaction Extended Abstracts. ACM, 2015, pp.71-72.

[18] M. Chen, Y. Zhang, Y. Li, et al. , "AIWAC: affective interaction through wearable computing and c10ud technology," IEEE Wireless Communications, vol. 22, no. 1, pp. 20-27, 2015.

[19] M. Ammi, V. Demulier, S. Caillou, et al., "Haptic Human-Robot Affective Interaction in a Handshaking Social Protocol," in

Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction. ACM, 2015, pp. 263-270.

[20] C. Lai, R. Hwang, H. Chao, M. Hassan, A. A1amri" A Buffer-Aware HTTP Live Streaming Approach for SDN-Enabled 50 Wireless Networks", IEEE Network, pp. 49-55, 2015.

19