Integration of the Humanoid Robot Nao inside a SmartHome: A Case Study

Athanasia Louloudi, Ahmed Mosallam, Naresh Marturi,Pieter Janse and Victor HernandezSchool of Science and TechnologyÖrebro University, Sweden 701 82

<first name>.<last name>h081@student.oru.se

April 26, 2010

AbstractThis paper presents a case study demonstrat-ing the integration of the humanoid roboticplatform Nao within a Network Robot System(NRS) application. The specific scenario of in-terest takes place in a smart home environment;the task being that of bringing a can of sodafrom a fridge to a human user. We use this con-crete scenario to evaluate how the performanceof such a robot can be affected by being embed-ded inside an intelligent domestic environment.This study points out that, by cooperating withdifferent components on the network the over-all performance of the robot is increased.Keywords: Network Robotics Systems, Do-mestic Robots, Nao robot, Smart home, PEIS-Ecology.

1 IntroductionA few years ago, the idea of living with robots,sharing everyday tasks and harmonically co-exist in the same environment, seemed to bea distant scenario. However, the use of suchtechnologies, aimed to inhabit our houses andhelp us with our everyday chores is not a dreamany longer. Ongoing research all over the worldindicates a trend to develop advanced roboticsystems aimed to the service of people in need.According to the International Federation of

Robotics, 7.1 million service robots for personaland private use were sold by the end of 2009 and11.6 million is anticipated to be sold by 2012.1The rapid growth in the field of robotics duringthe last decade provided a strong foundation forsmart homes and sensor networks.

A Smart home is a domestic intelligent en-vironment where various components such as afridge, oven, lights etc., are working togetherby exchanging information via the same localnetwork. The principal idea behind the smarthome concept is to use NRS techniques to in-tegrate different services within the home in aneffort to control and monitor the entire livingspace [1]. NRS can provide robot based ser-vices to improve care cost and the quality of lifein smart homes. These services are not realizedby a single stand-alone robot but by a combina-tion of different elements such as environmentalsensors, cameras, laser range scanners and hu-mans communicating and cooperating througha network.

In a stand-alone robot, all the sensorial andcomputational capabilities are self contained.In the context of a NRS, a stand-alone robot isperceived as part of the ecology itself [2], [3].Moreover, it can be beneficiated by the fluxof information coming from other devices con-nected to the same network, for example, a

1International Federation of Robotics, ExecutiveSummary of World Robotics 2009. See http://www.ifr.org/service-robots/statistics/

35

camera mounted to the ceiling can provide awider view than its own embedded cameras.Network robots are divided into three types:visible robots, unconscious robots, and virtualrobots [4], [5]. In this work we consider Naoas a visible robot since it has the role of be-ing the physical interface of the NRS inside thesmart home. Furthermore, according to Scopel-liti et al., the most preferred robots, are thosewhich are human-friendly in means of appear-ance, primarily resembling pets or toys [6].

This paper is organized as follows: First, insection 2, the problem formulation is presented,describing the overall tasks and the availabletools that endue this work. Then, in section 3we analyse some specific problems which arosein the implementation of our demonstration,emphasising in particular the contrast betweenour NRS approach and an alternative, singlerobot approach. The software architecture fol-lows in section 4. Here, the methodologies andstructure followed in order to handle this workare described. Finally concluding remarks closethis paper.

2 Problem FormulationConsider the following scenario:

Neils returns home from a long walk. Soonhe enters the living room he taps on the headof Tommy, a humanoid robot and rests on thesofa. Tommy perceives the request, offers tobring a refreshment to Niels and starts movingtowards the fridge in the kitchen. Whilewalking, it localizes itself to find its position inthe room. When Tommy enters the kitchen,it asks the fridge to open its door and use itsgripper to collect a soda can and bring it out.The robot identifies the requested drink, graspsit and returns to deliver the refreshment toNiels.

The above scenario can be decomposedinto the following simpler tasks that has to beperformed by the robot in order to fulfill theoverall goal:

1. Walk towards the fridge

2. Dock the fridge

3. Grasp the drink

4. Carry and hold the drink while walking

5. Deliver the drink to the user

These tasks can be grouped into three mod-ules which are localization, cooperative grasp-ing and the mobility module (explained in latersections of this paper).

The following paragraphs, introduce all theinformation needed in order to understand thecomponents that surround this project. We de-scribe the NRS infrastructure underlying oursmart home, and other details that constitutethe robotic platform Nao.

2.1 Available ResourcesThis section describes all the available toolsthat can be used in order to successfully ac-complish the overall task.

2.1.1 Test Environment

PEIS-Ecology:The concept of PEIS-Ecology, first introducedby Saffiotti and Broxvall in 2005 [7], is oneof the few existing realizations of the notionof network robot system. The name PEISstands for physically embedded intelligent sys-tems. PEIS can be defined as a set of inter-connected components, residing in one physi-cal entity which generalizes the notion of robot.Every component that is part of this ecology iscalled PEIS-component.

The PEIS Ecology model has been imple-mented in an open source middleware, calledthe PEIS Kernel to which all the PEIS compo-nents are linked. This PEIS-Kernel allows thecomponents to communicate and collaboratewith each other in a standardized way. Forcommunication, this middleware establishes apeer-to-peer network and performs dynamicrouting of messages between PEIS. All PEIScan cooperate using a uniform cooperationmodel, based on the notion of linking functionalcomponents: each participating PEIS can usefunctionalities from other PEIS in the ecology

36

in order to compensate or to complement itsown.

PEIS Home:The PEIS home2 is an experimental envi-ronment that looks like a typical bachelorapartment which was built to implement thePEIS-Ecology [8]. It consists of a living hall,bedroom and a small kitchen. The PEIS-Homeis equipped with communication and computa-tional infrastructure. This study is conductedinside the smart home to make use of variouscomponents of PEIS-ecology. Fig. 1 illustratesa few basic views of the home.

The PEIS components available for this workare:

• PEIS-FridgeThe PEIS-Fridge is a small sized refriger-ator with a motorized door, a camera andan attached robotic arm with a gripper.The camera takes images over a shelf inthe fridge and in combination with cluster-ing algorithms directs the gripper towardsthe soda cans. The fridge gripper is able tocollect the soda can from the interior spaceand is able to bring the drink outside.

• PEIS-CamA PEIS-Cam is a component that can pro-vide images from common 2D colour cam-eras. Several cameras are mounted in dis-crete areas of the house which can providea wide view of the living space.

• PEIS-PersonTrackingThis component is a tracking system con-nected to a set of stereo camera and normalmega pixel camera mounted on the ceiling,capable of tracking multiple persons.

2.1.2 Nao robot

Nao is a humanoid robot equipped with sonarsensors, 2 CMOS cameras and three-fingeredrobotic hands. It features a multimedia sys-tem with 4 microphones and 2 hi-fi speakers for

2The PEIS home is developed by the Center for Ap-plied Autonomous Sensor Systems (AASS). See http://aass.oru.se/~peis/

Figure 1: A rough sketch of the PEIS-Home(upperleft). A picture of the control deck (upper right).A picture of the kitchen with a fridge located un-der the workbench (lower left). A picture of thecommon room(lower right).

voice recognition and text-to-speech synthesis.The built-in functionalities of this robot such aslogo detection, mark detection using onboardcameras are loftily adjustable and are used forrobot localization. The robotic hands are usedfor grasping and holding small objects. Nao cancarry up to 300g using both hands. 3

Nao Robocup Edition has 21 degrees of free-dom (DOF) whereas Nao Academics Editionhas 25 DOF since it is built with two handswith gripping abilities. For this study the aca-demic version of Nao is used.

The Nao is based on Linux and it is a fullyprogrammable robot which uses its own frame-work called NaoQi. NaoQi, allows the devel-oper to access all the features and functional-ities of the robot through an Application Pro-gram Interface (API) which also provides theflexibility of executing tasks in sequential or-der, parallel and event based. The Integratedwireless network card of this robot can be usedto exchange information with other devices inthe network.

3Nao Documentation. Aldebaran Robotics. 2009.See http://academics.aldebaran-robotics.com

37

Figure 2: Software Architecture - Centralized andNRS approach

3 The Nao robot inside thePEIS-home

The first attempts to accomplish the sce-nario explained in section 2 were based onthe robot’s capabilities (centralized approachshown in Fig. 2), where all the perceptual in-formation of the environment and the tasks areperformed by one single agent. Consideringthat this humanoid platform is quite capable,it was expected to be able to perform success-fully the scenario’s tasks (localization, grasp-ing and navigation) but several drawbacks re-vealed through testing. These drawbacks werethe main reason for relying on external sourcesto fulfill the task. The first drawback has todo with localisation. The Nao is equipped withsonars, two cameras and a built-in logo recogni-tion ( “ALLandMarkDetection”), a vision mod-ule in which Nao recognizes special landmarkswith specific patterns on them. Our attemptsto leverage these sensory inputs for localisationfailed, due to the following reasons:

1. The unreliability of the internal mark de-tection algorithm led to missing the check-point (marks) on the predefined path.

2. Because of the wobbling of Nao while walk-ing the detection of the marks was unreli-

able.

3. Slight changes in the lighting conditions ofthe room, could lead to faulty detection ornot detection at all.

4. State of the art localisation using stereovision [12] cannot be implemented usingthe two onboard cameras since there is nooverlapping region between the data ac-quired by them.

The following section, describes the final im-plementation. The software architecture willbe described and a more detailed illustration ofthe basic three modules will ensue.

4 Software ArchitectureThe software architecture is shown in Fig. 2 andwas built having in mind the NRS approachwhich; as explained in [5], has the premise ofcombining robotics and information communi-cation technologies through a network in orderto realize a task or provide services to humanusers. The software architecture was developedusing Aldebaran’s NaoQi as a framework andcoded in C++. NaoQi gives access to all thefeatures of the robot, like sending commandsto the actuators, retrieving information fromthe robot memory, and managing Wi-Fi con-nections. This architecture facilates the inte-gration of new components without redesigningthe system architecture.

4.1 Control UnitThe control unit is built around a state ma-chine, which provides the logic to arbitrate theinteraction between the rest of the softwaremodules and determines the sequence of the ac-tions that should take place in order to achievethe goal described in the introduction of thispaper.

Fig. 3 shows the state diagram of the controlunit were each state represents an action to beexecuted and the transitions between states aredetermined by signals passed to the control unitby the rest of the modules.

38

Figure 3: State diagram for the control unit

Figure 4: Coordinate frame for the mobility mod-ule

4.2 The Mobility Module

The function of this module is to handle thenavigation of the Nao robot inside the PEIShome. This module receives as inputs the esti-mated current pose and the commanded posesthrough the controller module.

To represent the robots pose in the PEIShome, we have opted for the model suggestedin [9], where the degrees of freedom of the jointsof the legs and the feet of the robot are ignoredand the robot is represented as a rigid bodyoperating in a horizontal plane. The robot

Table 1: PSEUDO-CODE FOR MOBILITYMODULE

1 Retrieve the current pose from the lo-calization module.

2 Compute the angle AT between the de-sired pose and the current pose.

3 Compute the Euclidian distance DS be-tween the desired pose and the currentpose.

4 if AT > anglethresholdCommand Nao to turn AT degrees.

5 if (DS > Distancethreshold)Command Nao to move DS metersforward.

6 When Nao stops moving, repeat steps1 to 5 to reduce the error in the pose.

7 Wait for the next command from thecontrol unit.

pose(RP) is represented as follows:

R⃗P = [x, y, θ]T (1)

where x and y are the position in the coordinateframe shown in Fig. 4, and θ is the orientationof the robot with respect to the x axis.

The error between the current and the com-manded pose is decomposed in an orientationerror and a position error, that are later passedthrough the Nao API’s to orient the robot to-wards the target position and then to move itforward to correct the position error. However,the built in humanoid walking implementationof the Nao robot does not consider the externaldisturbances that affect the walking schema ofthe robot. This lack of feedback causes an errorbetween the desired pose and the physical poseof the robot. In order to reduce the impact ofthis error, the reported pose from the localiza-tion module is used to compensate the desiredpose of the robot.

The functionality of the mobility module canbe summarized in Table 1.

39

Figure 5: Robot detection and tracking usingthe background subtraction and color slicing algo-rithms

4.3 The Localization and Coordi-nation Module

As mentioned before the odometry error be-tween the desired and the actual pose of therobot is present due to external disturbancesthat affect the robot while walking. In order toreduce the impact of this error, we use an ad-hoc odometry algorithm that gives the robot afeedback about its actual position. Using al-ready present environmental monitoring cam-eras placed on the ceiling, background subtrac-tion [10] and color slicing algorithms [11], wecould efficiently localize the robot by detectingthe blue light on top of its head and providethe mobility module with the actual pose ofthe robot as shown in Fig. 5. We used thisalgorithm to calculate the robot position at al-ready known positions and built a lookup tablethat we used later to interpolate the robot po-sition. The pseudo code in Table 2, presentsthe basic steps of the algorithm. We did thatto decrease the effect of the noisy readings ofthe robot position due to the robot instabilitywhile walking.

The Localization module can be divided intothree steps:

1. Background subtraction

2. Color slicing

3. Position lookup table

Table 2: PSEUDO-CODE FOR LOCALIZA-TION AND COORDINATION MODULE

1 Capture background image B”2 Subtract the new frame from B.3 for each resulting region:

if area > areathresholdadd this region to ROI

4 for each ROI:for each pixel in the selected ROI

Calculate the Euclidian distancebetween this pixel and the colorthreshold C”.

if C > colorthresholdSet the pixel value to one

elseSet the pixel value to zero

5 The region with highest value will bethe robot pose R”.

6 Calculate the centroid for R and the re-sult will be single pixel P”.

7 The lookup table calculates the robotposition using P.

4.3.1 Background subtraction

In this step the algorithm subtracts each newimage from the background image and detectsthe regions of interest (ROI). ROI area hasto be greater than a certain threshold. Thethreshold has to be set manually according tothe ”tolerance” we want to give to our algo-rithm (a high value of threshold means thatonly objects with high area value is consideredinteresting, whereas a lower value of thresholdwill make the algorithm detect small objects asROI).

4.3.2 Color slicing

At this point, we consider only the ROI de-limited in the previous step. The ROI is pro-cessed in the RGB color space, where we calcu-late the Euclidean distance between the pixelin the ROI and a color of interest. If the dis-tance is less than a certain threshold, the pixelis classified as part of the robot.

40

4.3.3 Position lookup table

In this final step, we use a lookup table to re-turn the real robot pose. This lookup tableprovides a mapping from the robot’s positionin pixels to the real coordinates.

Although this ad-hoc solution is quite simple,the results obtained have significantly reducedthe error in the orientation in comparison withthe open loop walking schema.

4.4 The Cooperative GraspingModule

The robotic grasping is a very difficult problem,specifically grasping objects that are being seenfor the first time through vision. Solving thisproblem could be through using learning algo-rithm or building a 3-D model of the object ofinterests or by using both techniques. In thiswork we did not apply any learning or model-ing approaches, instead we assumed that a spe-cific object is handed in to the robot in a fixedpose. That means whenever the robot reachesthe right position for grasping it will start pre-defined movements in order to grasp with nofeedback. The grasping problem was simplifiedin order to tackle the problems of detecting thepresence of the object to be grasped and ap-proaching it. We installed a light source on topof the fridge gripper to be detected in the sameway as we detect the Nao robot head. The al-gorithm detects the robot and the object to begrasped and calculates the Euclidian distancebetween the robot and the object.

The walking module receives the distanceand commands the robot to move. After ap-proaching the grasping point, the robot doesthe following:

1. Command the fridge to open through thePEIS infrastructure

2. Command the fridge gripper to fetch a cer-tain drink out of the fridge

3. The robot docks the extended fridge grip-per to place itself in a position suitablefor grasping. This is done by indirect vi-sual servoing through the localisation of

Figure 6: Fridge door opening(upper left), fridgegripper bringing the can out(upper middle), trackedimage from the camera (upper right), Pre-definedrobot arm movements after reaching the desired po-sition for grasping the object(bottom left, bottommiddle and bottom right)

the gripper as perceived by the environ-mental camera

4. The robot then follows predefined arm andhand movements to grasp the object. SeeFig. 6

The algorithm is explained in Table 3.

4.5 HRI Module

The Human Robot Interface (HRI) module actsas the direct link between the user and the sys-tem. Through this module the user is able torequest the can of soda by tapping the robotshead.

This module will receive the current posi-tion (coordinates) of the human user from thePEIS person tracking system. These coordi-nates in turn will be passed to the mobilitymodule through the control unit as a final des-tination for the Nao robot.

The position of the user is determined by thismodule. See Fig. 2. When a centralized ap-proach is used, the task of finding the user isperformed by the robot itself (i.e. face recog-nition), while for the NRS approach, this taskis carried out by an external module (i.e. thePEIS person tracker).

41

Table 3: PSEUDO-CODE FOR COOPERATIVEGRASPING MODULE

1 Capture background image B”2 Subtract the new frame from B.3 for each resulting region:

if area > areathresholdadd this region to ROI

4 for each ROI:for each pixel in the selected ROICalculate the Euclidian distancebetween this pixel and the colorthresholds C1 and C2.

if C1 > colorthresholdset the pixel value to 1 for therobot region RROI

else if C2 > colorthresholdSet the pixel value to 1 for theobject region OROI

elseSet the pixel value to 0.

5 In RROI the region with highest valuewill be the robot pose R.

6 In OROI the region with highest valuewill be the object pose O.

7 Calculate the centroid for R and the re-sult will be single pixel “PR”

8 Calculate the centroid for R and the re-sult will be single pixel “PO”

9 The lookup table calculates the robotposition and the object position usingPR and PO.

10 Calculate the Euclidian distance andcommand the robot to approach thisposition.if (Destinationreached)

Start the predefined movements ofthe arms.

5 System DemonstrationThe system was evaluated using two differentsetups where, inspired by the scenario depictedin section 2, the robot was commanded by theuser to follow a predefined path (from the livingroom to the kitchen) in order to collect andbring back a known object placed in the fridgegripper. Twenty rounds for each setup were

performed.In the first setup, it was required to make

the robot locate itself inside the environmentwithout being assisted by any other componentinside the PEIS home. We opted for using theon-board vision capabilities of the robot to de-tect pre defined marks (i.e. Nao Landmark De-tection) as shown in Fig. 7 and the marks wereplaced along the predefined path from the liv-ing room to the kitchen and the robot was ex-pected to use them to correct its position andorientation in order to complete one test round.Poor results were obtained due to the noise in-troduced from embedding the localization mod-ule inside one single agent (in this case, theNao robot), such as the wobbling produced bythe walking or in some cases, when the robotmissed one mark while moving, the error in thelocalization was increased in a way that wasalmost impossible to find the right path to suc-cessfully complete the task. This drawbackslead to a successful rate of only 20% of therounds.

For the second setup, the robot was assistedby the NRS as described in section 4.3. Itwas observed that the performance was sub-stantially improved since all the rounds weresuccessfully completed and as a final demon-stration of the system, a person who is not fa-miliar with the technical details of this studywas taught how to operate and interact withthe system.4 Once again the task was success-fully completed. The drawback of this setupis that the system is heavily dependent on thenetwork stability. A failure in the connectionor missing information from any of the involvednetwork components may negatively affect thefinal output.

6 ConclusionsIn this work we have successfully integrated ahumanoid robot into a NRS to solve a problemin a common day scenario; bringing a can ofsoda to a human user. This problem involvesseveral tasks such as localization, mobility and

4Integrated Project Work demos. See http://www.youtube.com/user/loutfiamy

42

Figure 7: Robot localization using land marksand Land mark detected by Nao(in sight).

cooperative grasping which has to be accom-plished in order to successfully achieve the over-all goal.

While the Nao is a capable platform; due tothe complexity of the problem to be solved andthe challenges that home robotics implies, weconclude that the capabilities of the robot canbe enhanced and the complexity of the problemcan be reduced by decomposing it in simplertasks executed in cooperation with specializedcomponents in the NRS (i.e. the PEIS Ecol-ogy).

The NRS brings also expandability to theNao itself when new components are added tothe network and Nao interacts with them tosolve a novel task. The NRS is also beneficiatedwith the incorporation of Nao, since it becamea visible robot that provides different servicesand acts as an interface between the user andthe PEIS Ecology.

Future work may include how to exploit thesensorial capabilities of the Aldebaran’s Nao,such as voice and face recognition to allow amore user friendly interface between a smarthome and the human user who is request-ing different services from the ubiquitous net-work. This work can also be expanded by usingplanning and searching techniques to allow therobot to move in non predefined paths and theuse of the robot’s internal sonars can be used

to allow obstacle avoidance.

7 AcknowledgmentsThis work was supported by AASS (AppliedAutonomous Sensory Systems, Örebro Univer-sity, Sweden) laboratories through the courseIntegrated Project Work (IPW 2009). Manythanks to all the people at the AASS labora-tories for their contributions to this work, es-pecially Dr. Federico Pecora, Dr. Dimitar N.Dimitrov and Todor Stoyanov.

References[1] A. Sanfeliu, N. Hagita, and A. Saffiotti,

“Network robot systems,” Robotics andAutonomous Systems,Elsevier 2008, 2008.

[2] N. Kubota and K. Nishida, “Coopera-tive perceptual systems for partner robotsbased on sensor network,” IJCSNS, Inter-national Journal of Computer Science andNetwork Security, vol. 6, no. 11, pp. 19–28,2006.

[3] K. Lynch, I.B.Schwartz, P. Yang, andR. Freeman, “Decentralized environmen-tal modeling by mobile sensor networks,”Robotics, IEEE Transactions on, vol. 24,pp. 710–724, June 2008.

[4] K. Kemmotsu, T. Tomonaka, S. Shiotani,Y. Koketsu, and M. Iehara, “Recogniz-ing human behaviors with vision sensorsin a network robot system,” in Proceed-ings - IEEE International Conference onRobotics and Automation, vol. 2006, (Or-lando, FL, United states), pp. 1274 – 1279,2006.

[5] T. Akimoto and N. Hagita, “Introductionto a network robot system,” in 2006 Inter-national Symposium on Intelligent SignalProcessing and Communications (IEEECat. No.06EX1444), pp. 91 – 4, 2006.

[6] M. Scopelliti, M. Giuliani, A. D’Amico,and F. Fornara, I had a robot athome…Peoples’ representation of domestic

43

robots. In S. Keates, J. Clarkson, P. Lang-don and P. Robinson, Designing a moreinclusive world. London: Springer Verlag„2004.

[7] M. Broxvall, M. Gritti, A. Saffiotti,B. Seo, and Y. Cho, “PEIS ecology: Inte-grating robots into smart environments,”in Proc. of the IEEE Int. Conf. onRobotics and Automation (ICRA), (Or-lando, FL), pp. 212–218, 2006. Online athttp://www.aass.oru.se/˜asaffio/.

[8] A. Saffiotti and M. Broxvall, “PEISecologies: Ambient intelligence meetsautonomous robotics,” in Proc of theInt Conf on Smart Objects and Ambi-ent Intelligence (sOc-EUSAI), (Grenoble,France), pp. 275–280, 2005. Online athttp://www.aass.oru.se/˜asaffio/.

[9] R. Siegwart and I. R. Nourbakhsh, In-troduction to Autonomous Mobile Robots.Bradford Book, 2004.

[10] J. Davis and V. Sharma, “Fusion-based background-subtraction using con-tour saliency,” in CVPR ’05: Proceed-ings of the 2005 IEEE Computer Soci-ety Conference on Computer Vision andPattern Recognition (CVPR’05) - Work-shops, (Washington, DC, USA), pp. 162–182, IEEE Computer Society, 2005.

[11] R. Gonzalez and R.E.Woods, Digital Im-age Processing. Pearson Prentice Hall,2008.

[12] Don Murray and James J. Little, “UsingReal-Time Stereo Vision for Mobile RobotNavigations,” Auton. Robot, vol. 8, no. 2,pp. 161–171, 2000.

44