PROJECT REPORT ON

Gesture recognition by a 3d camera and user impersonation via a

humanoid

DEPARTMENT OF ELECTRONICS AND INSTRUMENTATION ENGINEERING

INSTITUTE OF TECHNICAL EDUCATION AND RESEARCH SIKSHA ‘O’ ANUSANDHAN UNIVERSITY, ODISHA

SESSION: 2008-12

PROJECT REPORT ON

GESTURE RECOGNITION BY A 3D CAMERA AND USER

IMPERSONATION VIA A HUMANOID

SUBMITTED IN PARTIAL FULFILLMENT FOR THE AWARD OF BACHELOR’S DEGREE IN ELECTRICAL AND ELECTRONICS ENGINEERING

SUBMITTED BY PIYUSH ROUTRAY 0811015112

SUBRAT RATH 0811014022 SANKALP MOHANTY 0811014025 GHANSHYAM BHUTRA 0811014039

UNDER THE GUIDANCE OF

ASST. PROF. FARIDA ASHRAF ALI ASST. PROF. RESHMA BHOI

DEPARTMENT OF ELECTRONICS AND INSTRUMENTATION ENGINEERING

INSTITUTE OF TECHNICAL EDUCATION AND RESEARCH SIKSHA ‘O’ ANUSANDHAN UNIVERSITY, ODISHA

SESSION: 2008-12

DEPARTMENT OF ELECTRONICS AND INSTRUMENTATION

ENGINEERING

Certificate

This is to certify that the thesis entitled “ Gesture Recognition by a 3D

camera and User Impersonation via a Humanoid ” submitted by PIYUSH

ROUTRAY has completed his major project, in the academic year 2011-2012

for the partial fulfillment of Bachelor of technology under Siksha ‘O’

Anusandhan University, under the department of Electronics and

Instrumentation Engineering, of Institute of Technical Education & Research,

Bhubaneswar.

Date: 16th

April, 2012

Prof. G.S.C. Mishra Asst. Prof F. A. Ali

HOD (E&I) Project Guide

INSTITUTE OF TECHNICAL EDUCATION AND

RESEARCH BHUBANESWAR

SIKSHA ‘O’ ANUSANDHAN UNIVERSITY BHUBANESWAR

DECLARATION

WE DECLARE THAT EVERY PART OF THE PROJECT REPORT SUBMITTED IS

GENUINELY OUR OWN WORK AND HAS NOT BEEN SUBMITTED FOR THE AWARD

OF ANY OTHER DEGREE. WE ACKNOWLEDGE THAT, IF ANY SORT OF

MALPRACTICE IS DETECTED IN RELATION TO THIS REPORT, WE SHALL BE HELD

LIABLE FOR IT.

PIYUSH ROUTRAY

0811015112

SUBRAT RATH

0811014022

SANKALP MOHANTY

0811014025

GHANSHYAM BHUTRA

0811014039

INSTITUTE OF TECHNICAL EDUCATION AND

RESEARCH BHUBANESWAR

ACKNOWLEDGEMENT

WE WISH TO IMPART OUR HEARTFELT GRATITUDE AND SINCEREST

APPRECIATION TO THE FOLLOWING PERSONS WHO IN ONE WAY OR ANOTHER

HAVE CONTRIBUTED AND SHARED THEIR IMMEASURABLE ASSISTANCE TO THE

SUCCESSFUL COMPLETION OF THIS PROJECT REPORT:

TO Dr. G.S.C. MISHRA, HOD, E&I DEPARTMENT;

TO Mrs. FARIDA ASHRAF ALI, PROF., PROJECT GUIDE, E&I DEPARTMENT;

ABOVE ALL, TO ALL THE FACULTY MEMBERS OF OUR DEPARTMENT AND

OUR FRIENDS, FOR THEIR SUPPORT.

PIYUSH ROUTRAY

0811015112

SUBRAT RATH

0811014022

SANKALP MOHANTY

0811014025

GHANSHYAM BHUTRA

0811014039

LIST OF FIGURES

Fig No. Description Page No.

1.1 Process Overview 6

1.2 Complete Work Flow 6

3.1 Difference between Legged and Wheeled Robot 12

3.2 SolidWorks Software 14

3.3 Individual structure of Humanoid 15

3.4 Complete Humanoid 16

3.5, 3.6 Adams Software Model 17

3.7 Basic Components used 18

3.8 Arm with Gripper 18

3.9 Complete Upper Body 19

3.10 Two Arms with Circuit Boards 19

3.11 Actual Model vs. Virtual Model 20

4.1, 4.2 KINECT Sensor 22, 23

4.3 Pin Diagram of ATMEGA 16 27

4.4 AVR with USART 29

4.5 XBEE Pinouts 30

4.6 XBEE Specification 31

5.1 Simulink Model 37

5.2 Pulse for Servo 40

6.1 RGB Image 42

6.2 IR Image 43

6.3 Skeletal Image 45

6.4 Calculation of Link Length 47

6.5 Calculation of Angles 49

6.6, 6.7,

6.8

Graphs for Torque Profile of Motors 50

6.9 Synchronization between User and Virtual Model 51

ABSTRACT

ON

GESTURE RECOGNITION BY A 3D CAMERA AND USER

IMPERSONATION VIA A HUMANOID

The central idea is being able to analyze 3D images in real time domain and apply similar techniques

available for 2D images like face recognition, gesture recognition, etc. The idea has many potential

real life based applications including entertainment, domestic, industry and etc. We aim at creating a

virtual and a hardware model which would have several practical applications of engineering in every

field possible.

The basic principle of working of the project is based on first building a complete mechanical model

and its assembly. Then the model is integrated with multidisciplinary variables like motion,

acceleration, torque, friction, etc. Then the model is controlled via Real Time parameters which are

given in by the sensor. The final output is complete simulation of the virtual model and the humanoid

using gesture movement of the user.

Name: Subrat Rath - 0811014022

Sankalp Mohanty - 0811014025

Ghanshyam Bhutra - 0811014039

Piyush Routray - 0811015112

CONTENT

1. INTRODUCTION....................................................................................................... 1

1.1 Humanoid……………………..................................................................................... 2

1.2 Communication............................................................................................................ 3

1.3 KINECT Sensor........................................................................................................... 3

1.4 Servo Motor ………………........................................................................................ 4

1.5 Aim of the Project…………........................................................................................ 5

2. LITERATURE REVIEW…….………………………………………… …............. 7

3. DESIGN OF HUMANOID..….………………………………………… ….............11

3.1 Why Humanoid?……………..................................................................................... 12

3.2 Calculation of Dimensions...…...................................................................................1 3

3.3 Software used for Designing - SolidWorks................................................................ 14

3.4 Virtual Design in SolidWorks.....................................................................................14

3.5 MSC – Adams……………….....................................................................................17

3.6 Hardware Model……………..................................................................................... 18

3.7 Actual (Hardware) Model vs. Virtual (Software) Model…....................................... 20

4. CONTROL ARCHITECTURE………………………………………… ….............21

4.1 KINECT Sensor ……………..................................................................................... 22

4.1.1 Using Kinect................................................................................................ 24

4.1.2 Kinect Drivers.............................................................................................. 24

4.2 ATMEL AVR ……………........................................................................................ 25

4.2.1 Atmega 16..................................................................................................27

4.2.2 Pin Description.......................................................................................... 28

4.2.3 Using the USART of Atmega 16.....................................................,.........28

4.3 ZIGBEE Communication…...................................................................................... 30

4.3.1 XBEE……..................................................................................................30

4.3.2 XBEE Working Principle...........................................................................31

5. PROCESS INVOLVED……………….……………………………...................... 32

5.1 Data Points from KINECT…................................................................................... 33

5.1.1 Calculation of Joint Angles....................................................................... 34

5.1.2 Calculation of Link Lengths...................................................................... 34

5.2 Denavit-Hartenberg Methodology............................................................................ 35

5.2.1 Simulink Model……………..................................................................... 37

5.3 Modes of Operation of ZIGBEE...............................................................................38

5.4 Memory Addressing Technique................................................................................39

5.5 Controlling the Servo Motor….................................................................................40

6. CODES IMPLEMENTED……………………………………….……......………..41

6.1 Coding in KINECT………………………………………………………………….42

6.1.1 RGB Image……………………………………………………………. ….42

6.1.2 IR Image……………………..................................................................….43

6.1.3 Skeletal Image……………………………………………………………..44

6.1.4 Finding out the link lengths………………………………………………..45

6.1.5 Finding out the angles…..………………………………………………..47

6.2 Coding in ADAMS – MATLAB….……………………………………….………. 50

6.3 Coding in Microcontrollers…………………………………………………………52

7. CONCLUSION ………………………………………….………………………...55

8. BIBLIOGRAPHY..……………………………………….………………………...56

1

CHAPTER 1

INTRODUCTION

Going by the present scenario, it can be very well noticed that not in the very distant

future, humanoids seem to be the best alternative for human labour. The present economy of

labour all around the globe is pushing this issue at a tremendous rate. Introduction and

development of robots namely NAO, ASIMO, etc in the market shows us the rapid

development in the field of Humanoid robotics technology. Machines are devices that make

the work of a man easier. Ranging from large scale machines like car crushers to small scale

machines like staplers, it is classified into diverse groups. Similarly, Humanoids are machines

built and programmed for a dedicated purpose, the only difference being the fact that it

resembles a human being.

2

1.1 Humanoid

Going by the dictionary meaning, Humanoid means “having an appearance or character

resembling that of a human ". It can be said that humanoids are complex machines resembling

a human in terms of size, shape, degree of freedom, etc. The boundary of application for

humanoids is shared by human themselves. The motivation for studying humanoid robots in

particular arises from diverse sociological and commercial interests ranging from the desire to

replace humans in hazardous occupations (de-mining, nuclear power plant inspection,

military inventions, etc.), to the restoration of motion in the disabled(dynamically controlled

lower-limb prostheses, rehabilitation robotics and functional neural stimulation). With the

advent of new methods, it can adapt to more and more complicated environments and even

surpass human at speed and accuracy.

Locomotion, the ability of a body to move from one place to another is a defining

characteristic of animal life. It is accomplished by manipulating the body with respect to the

environment. In the natural setting, locomotion takes on many forms, whether it's the

swimming of amoebas, flying of bits, or walking of humans. The diversity of animal

locomotion is truly astounding and surprisingly complex. The same is true in objects crafted

by man: air-planes have wings that create lift for flight, tanks have tracks for traversing

uneven terrain, and automobiles have wheels for rolling efficiently- and robots are now

walking on their own two legs!

The present day humanoids move and work with a predefined set of instructions and motions

to perform a specific task. An important and current topic of discussion revolves around

gesture imitation. Imitation refers to the act of impersonating a user's actions, be it hand

movement or leg movement. Achieving perfect imitation of a human requires tremendous

calculations and harmonious work of many different aspects of the system. As we go on to

develop more and more complicated control algorithms, better and more efficient functional

ties can be realised. Building, designing and working with practical robots requires

3

knowledge in three major engineering fields: mechanical, electrical and software. These days,

mobile robots are being called on to perform more and more complex and practical tasks.

Autonomous robots place special demands on their mobility system because of the

unstructured and highly varied environment the robot might drive through and the face that

even the best sensors are poor in comparison to a human's ability to see, feel and balance.

This means that the mobility system of a robot that relies on those sensors will have much

less info about the environment and will encounter obstacles that it must deal on its own. The

best mobility system for a robot to have is one that effectively accomplishes the required task;

without regard to how well a human course uses it. A common form of mobile robot today is

semiautonomous, where the robot has some sensors and acts partially on its own, but there is

always a human in the control loop through a link i.e. Telerobotic. We are using the technique

for a teleoperated robot - where there are no sensors on the bot that it uses to make decisions.

1.2 Communication

Next topic of focus is communication. It is a process by which two or more entities can

exchange or share information. Communication is done through a channel or a path which

may or may not have a physical existence. Communication between two or more points

which are not physically connected is called wireless communication. Communication in

wireless can be of few meters such as remote control operation or as far as millions of

kilometres such as deep space radio communication. Wireless communication is generally

done via Radio frequency or by microwave or by infrared. Communication can be from point

to point, point to multipoint or broadcasting. In this project, for wireless communication,

Zigbee and Xbee are implemented. ZIGBEE is a wireless technology developed for

applications mostly based on low cost, low power and wireless sensor networks. It follows

ZigBee protocol for communication which carries all the benefits of the 802.15.4 protocol

with added networking functionality. It can operate in the frequency ranges of 2.4-2.484GHz,

902-928MHz and 8680-868.6MHz. ZigBee protocol was developed by Zigbee Alliance. The

key feature of ZigBee protocol is that it supports mesh networks. The hardware devices which

have been used for ZigBee communication are Xbee trans-receivers. They work within a

range of 300meters and require a supply voltage of 2.8-3.4 volts. They are cheap, robust and

consume very less power.

4

1.3 KINECT Sensor

Sensor is a converter that measures any physical quantity and converts it into a signal which

can be read by an observer or by an instrument. Here the sensor used is a camera. Basically

camera is a device that records and stores images. These work in the visible spectrum of the

EM spectrum. The traditional cameras operate in a two-dimensional space i.e. the images are

viewed as a planar representation of the space we are in. But with the advent of technology,

there are techniques which are used for 3-D imaging. It creates and enhances the illusion of

depth in an image. These days, with the use of image processing and colour perception,

sensors which can provide a 3D experience have been developed. A rather significant

development can be seen with KINECT sensor which is a motion sensing device developed

by Microsoft for the XBOX 360 gaming console which is used in this project. It employs the

principle of Gesture Recognition of the user‟s movements. Gesture is simply any posture of

the user with some intention. It is far easier to point to an object rather than verbally describe

its exact location. Gestures are recognized in by mapping of individual camera images into

MATLAB and are compared to predefined conditions.

1.4 Servo Motor

For realization of gestures via user impersonation, a model is required. The model‟s major

component is a Servo Motor. It is a motor which forms part of servomechanism which is a

device that uses error-sensing negative feedback to correct the performance of a mechanism.

The motor is paired with an encoder to provide position/speed feedback. Servos are controlled

by sending a PWM (Pulse Width Modulation) signal i.e. a series of repeating pulses with

variable width. Generally servos are connected through a standard three-wire connection: two

wires for DC power supply and one for control, carrying the pulses. The signal is easily

generated by simple electronics or by microcontrollers. The angle of movement is determined

by the duration of pulse applied to the control wire. When these servos are commanded to

move, they will move to the position and hold it. If an external force pushes against the servo

while it is holding its position, it will resist from moving from that position. The maximum

5

amount of force the servo can exert depends on the torque rating of it. Moving on to the

model‟s description, there are 18 DOF (Degrees of Freedom), the complete description of

which has been illustrated in the following pages.

1.5 Aim of the Project:

Model a humanoid (Software and Hardware).

Embed the characteristics of a human being into it.

Take input from a 3D camera.

Control the humanoid with the Real Time input variables from the camera.

6

Figure 1.1

Figure 1.2

7

CHAPTER 2

LITERATURE REVIEW

Presented in this chapter are reviews about some of the related scientific papers,

journals and published IEEE papers.

Vitor M. F. Santos, Filipe M. T. Silva. “Engineering Solutions to Build an

Inexpensive Humanoid Robot Based on a Distributed Control Architecture”.2005 5th

IEEE-RAS International Conference on Humanoid Robots.

This paper presents the main steps to design a low cost fully autonomous humanoid

platform and the set of solutions proposed, thus being similar to most of our mechanical

modeling requirements. The main scope of the project beneath this paper is to carry out

research on control, navigation and perception, whilst offering opportunities for under and

pos-graduate students to apply engineering methods and techniques. The main features of the

22 degrees-of-freedom robot include the distributed control architecture, based on a CAN

bus, and the modularity at the system level. Ours being a 18DOF bot, accepts most

inspiration from this project. Although some issues are yet to be addressed, the stage of

development is already mature for practical experiments and to obtain the first conclusions on

the potential of the proposed solutions.

8

Milton Ruas, Filipe M. T. Silva, Vitor M. F. Santos. “A Low-Level Control

Architecture for a Humanoid Robot”.

This paper proposes general low-level control architecture for a small-size humanoid

robot using on off-the-shelf technologies. The main features of this implementation are the

distributed control approach and the relevance given to the sensorial information. Some

practical issues on servomotor control are given since that turned out necessary before

entering higher levels of control. Particular attention is given to the low-level control of RC

servomotors and the enhanced performance achieved by software compensation. The

distributed set of controller units is the key element towards a control system that

compensates for large changes in reflected inertia and providing position and velocity

control. Furthermore, a kind of intermediate level control is implemented as a local controller

based on force sensing, providing robust and adaptive behavior to changes in a slope surface.

ElvedinKljuno, Robert L.Williams II. “Humanoid Walking Robot: Modeling, Inverse

Dynamics, and Gain Scheduling Control”. Hindawi Publishing Corporation Journal of

Robotics.

This article presents reference-model-based control design for a 10 degree-of-freedom

bipedal walking robot, using nonlinear gain scheduling. The main goal is to show

concentrated mass models can be used for prediction of the required joint torques for a

bipedal walking robot. Relatively complicated architecture, high DOF, and balancing

requirements make the control task of these robots difficult. Although linear control

techniques can be used to control bipedal robots, nonlinear control is necessary for better

performance. The emphasis of this work is to show that the reference model can be a bipedal

walking model with concentrated mass at the center of gravity, which removes the problems

related to design of a pseudo-inverse system. Another significance of this approach is the

reduced calculation requirements due to the simplified procedure of nominal joint torques

calculation. Kinematic and dynamic analysis is discussed including results for joint torques

and ground force necessary to implement a prescribed walking motion. An inverse plant and

a tracking error linearization based controller design approach are described. A novel

combination of a nonlinear gain scheduling with a concentrated mass model for the MIMO

bipedal robot system is proposed.

James Kuffner, Koichi Nishiwaki, Satoshi Kagami, Masayuki Inaba, Hirochika Inoue.

“Motion Planning for Humanoid Robots”. 11th Int‟l Symp. of Robotics Research

(ISRR 2003)

This paper dwelled upon prospects of biped humanoids as back as 2003! In order to

improve the autonomy and overall functionality of these robots, reliable sensors, safety

mechanisms, and general integrated software tools and techniques are needed. We believe

that the development of practical motion planning algorithms and obstacle avoidance

software for humanoid robots represents an important enabling technology. This paper gives

an overview of some of our recent efforts to develop motion planning methods for humanoid

robots for application tasks involving navigation, object grasping and manipulation, footstep

9

placement, and dynamically-stable full-body motions. Most primitive techniques for

humanoid robot development are discussed in this paper.

Stefan Waldherr, Roseli Romero, Sebastian Thrun. “A Gesture Based Interface for

Human-Robot Interaction”2000 Kluwer Academic Publishers.

The thesis project basically dwells with gesture recognition of service robots. Much has not

been discussed about the mechanical structure, rather concentrates on image processing. It

specifically divided the job in „tracking‟, „gesture recognition‟ and „integration‟. Tracking

involves following the operator through the office or hall and characterising by face and shirt

colour. The shirt colour is assumed uniform and chromatic colour tracking is performed to

minimise the colour variation in operators‟ skin. This also prevents any influence due to

variation in lighting condition. Gesture recognition or pose analysis involves acquiring arm

poses from the mounted camera and then, analysis of arm poses for inferring meaning or

temporal template matching. The temporal template matcher integrates the multiple images

and then compares with its database. Dynamic and static response system in the process thus

involved is verified through rigorous experimentation. Integration deals with ordering the

robot to do the task, in this case, cleaning of office place. Different poses of arms tells the

robot to perform the intended task.

Ralf Salomon, John Weissmann. “Gesture Recognition for Virtual Reality

Applications Using Data Gloves and Neural Networks”.

This paper does not involve image processing. It rather uses electronic signals generated from

a hand glove to interpret expectations from the robot. The basic idea is to have different

permutations of angles presented by 18 sensor points in a single palm. The angles of two

joins in a finger, between fingers, thumb rotation, palm arch, wrist pitch and wrist-yaw form

nodes. While processing, nodes are interconnected and compared with previously stored hand

geometry data. However, the paper has very little experimental results and thus the technique

has to be verified.

David Kortenkamp, Eric Hubber, R. Peter Bonasso. “Recognising and interpreting

gestures on mobile Robot”. Robotics and Automation Group, NASA

The paper presents a very similar work to our area of work. They have presented work on

real-time, 3D gesture recognition system and mounted it over mobile robot. However, the

robot is not a humanoid, but a wheeled system. The robot is intended to interpret the orders

and not just following them. The pattern understanding mechanism adopted is proximity

space method. In this method the processor searches for the head of the operator, and then

subsequently links shoulder, upper arm and fore arm to the head via three links. The

movement vectors thus produced are analysed and proximity space is inferred. The method

acquires and reacquires the images override actions for inconsistent movement vectors.

Reactive action packaging specifies how and when to carry routine procedures through

conditional sequencing. The use of 3D image helps to perform the exact task out of wide

range of possible interpretations of corresponding images.

10

Mark Becker, EfthimiaKefalea, EricMael,Christop Von DerMalsburg, MikePagel, J

Triesch,Jan C. VorbrUggen, Rolf P. Wurtz, StefanZadel.GripSee: “A Gesture-

controlled Robot forObject Perception and Manipulation”

GripSee is a small step towards the bigger goal of preparing service robots. It

basically deals in improving the perception quality of the present robot system. However, the

paper concentrates on work limited to a particular environment only. However, there remains

requirement of high spatial precision for high resolution colour sensitivity demands.

Developed in C++ FLAVOR, the programming directs a dedicated computer for image

processing as well as another for arm movement and functioning. A particular signal of two

fingers directs the robot to the object it has to grip. The image of the object is analysed in

order to calculate the best possible surface to capture using the two gripping mechanisms of

the robot. It follows a predefined hierarchy of hand tracking, hand posture analysis, object

fixation, object recognition and grip planning before trajectory planning and gripping the

object. Similarly, it tracks the hand, analyses the posture(s) and then drops the object.

Christian Plagemann, VarunGanapati, Daphne Koller, Sebastian Thrun. “Real-time

identification and Localization of Body parts from Depth Images” 2010

www.hindawi.com

In this research, after acquiring the image through depth detection stereo vision, the

processor has to further process the image to undermine the action. The interest point method

identifies geodesic extrema on the surface mesh and then 3D vector orientation takes place.

The range of data thus acquired, facilitates segmentation of human body from the background

structure and also disambiguated visually similar poses. Real time high frame rate calculation

module prevents human-robot interaction from suffering any undue time delays. The research

also aims at maximising the repeatability of the detector, so that likelihood of detecting a

once-detected interest point is high. It has also discussed AGEX interest point method and

sliding window approach for identification of parts.

Luciano Spinello, Kai O. Arras, Rudolph Triebel, Roland Siegwart. “A layered

Approach to people detection in 3D Range Data”. 2010 www.aaai.org

Taking into account, the drastic changes in body pose, distance from sensor, self-

occlusion and occlusion by other objects, this research work subdivides a person into parts

defined by different height levels, and uses highly specialised classifiers for each part. It does

not necessarily choose ground as the reference frame, but rather tracks the person excluding

the background, whatsoever. The segmentation is done on point cloud segmentation manner

and then segments are rearranged to create the image. AdaBoost machine learning algorithm

approach is used in the process.

11

CHAPTER 3

DESIGN OF HUMANOID

Being inspired by the biology around is perhaps the best way to develop a

mechanical machine to cater to the needs of its humane masters. The greatest requirement in

the development procedure of the present day robots is perhaps the effective manoeuvring

skill. However attractive it may look to copy but even the simplest of near perfect biological

process are too complex to emulate. Considering various options (discussed later), the subject

of the project was chosen to be HUMANOID.

12

3.1 Why Humanoid?

Wheeled robots might be the easiest to accomplish considering numerous factors such

as low cost, easier electronic circuits, simplest geared dc motors, rugged structure and

unmatchable stability. However, certain tracks of manoeuvre demand finely engineered

legged robots, as:

Figure 3.1

13

3.2 Calculation of Dimensions:

The average height of a human being is 5 feet 3 inches. When conceiving a humanoid

platform countless decisions have to be made. Specifications must be defined and applied to

improve limits both on skills and overall objectives .In what ways the physical and functional

requirements are related to the robot dimensions, the mobility skills and performance of the

selected tasks like walking, turning, kicking a ball etc are predetermined. There are many

factors such as constraints in dimension, economic considerations, physical tangibility, final

objective or goal, limitation in availability of resources, which have a huge impact in

determining the robot‟s structure for construction. For the dimensioning of the humanoid

following steps have been followed:

1. First the height of the humanoid was decided based on the objective of the robot.

2. A scaling ratio was found out for the humanoid.

3. The human body part lengths were scaled down using the scaling ratio and finally link

lengths of the humanoid were found out.

The height was decided to be about 80cm and using the scaling ratio the individual

link lengths were obtained as follows:

LINK NAME LENGTH (in mm)

Upper arm 140

Lower arm 125

Upper leg 252

Lower leg 185

Body 250

Feet 85

Table 3.1

After the structure height (800mm) and remainder body proportions, the next issue

that comes is the determination of degrees of freedom (DOF). As per the end objective of the

project and complications in the structure, it was decided to have 18 degrees of

freedom(DOF) for the humanoid.

14

3.3 Software used for Designing –SolidWorks:

Solid Works is a 3D mechanical CAD (computer-aided design) program that runs on

Microsoft Windows and is being developed by Dassault Systèmes SolidWorks Corp., a

subsidiary of Dassault Systèmes. SolidWorks is a Parasolid-based solid modeler, and utilizes

a parametric feature-based approach to create models and assemblies.

Figure 3.2

3.4 Virtual Design in SolidWorks:

The model was first divided into different links. Every link was modeled individually and

then finally all the links were mated so as to form a complete model.

The links include:

Body.

Shoulder Joint.

Upper Arm.

Lower Arm.

Pelvis Joint.

Upper Leg.

Lower Leg.

Feet.

15

Figure 3.3

BODY

UPPER ARM

SHOULDER

JOINT

LOWER ARM

LOWER LEG

UPPER LEG

FEET

16

COMPLETE BODY

Figure 3.4

17

3.5 MSC – Adams:

Adams multi-body dynamics software enabled us to easily create and test virtual

prototypes of mechanical systems in a fraction of the time. Unlike most CAD embedded

tools, Adams was used to incorporate real physics by simultaneously solving equations for

kinematics, statics, quasi-statics, and dynamics. We also found how to import MATLAB

code into the Adams model. Using this software, all the DOFs were enabled in the virtual

model. The model was configured to take input from MATLAB and the plant was exported

which resulted in a „ .m „ file for use in MATLAB.

Figure 3.5

Figure 3.6

18

3.6 Hardware Model:

After completion of solid work 3D models of individual parts, the next step was to

construct the real hardware models. Various decisions had to be made regarding the choice of

material to be used for construction and thus aluminium was chosen as it light as well as

strong. The whole structuring of humanoid was done keeping in mind that motion between

the joints should be frictionless and the net weight of the upperbody should be lesser than the

lower body. L shaped aluminium channels were used,which gave immense strength to the

inner layers. Lock nuts were used so as to secure the joints.

Finally the servo motors rated 17kgcm were implanted in the humanoid and metallic

arms were used for coupling.

Figure 3.7

Figure 3.8

19

Figure 3.9

Figure 3.10

20

3.7: Actual (Hardware) Model vs. Virtual (Software) Model

Figure 3.11

This shows a comparison between the actual and the virtual model. In this image, the motors

except for the gripper have not been fixed. Due to the lack of resources and time, the working

is going to be demonstrated using just the upper-body parts and the locomotion via wheeled

bot.

21

CHAPTER 4

CONTROL ARCHITECTURE

The control architecture is the suite of control required for the behavioral

synchronization between human and humanoid robot. For the convenience of study, this part

has been divided into three parts, first section is about Kinect Sensor, and the second section

includes the AVR controllerand the third section contains the information about Zigbee

communication.

22

4.1 KINECT Sensor:

Kinect, originally known by the code nameProject Natal is a motion sensinginput device by

Microsoft for the Xbox 360video game console. Based around a webcam-style add-on

peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox

360 without the need to touch a game controller, through a natural user interface using

gestures and spoken commands.Kinect was launched in North America on November 4,

2010,in Europe on November 10, 2010, in Australia, New Zealand and Singapore on

November 18, 2010, and in Japan on November 20, 2010.

Kinect builds on software technology developed internally by Rare, a subsidiary of Microsoft

Game Studios owned by Microsoft, and on range camera technology by Israeli developer

PrimeSense, which developed a system that can interpret specific gestures, making

completely hands-free control of electronic devices possible by using an infrared projector

and camera and a special microchip to track the movement of objects and individuals in three

dimension. This 3D scanner system called Light Coding employs a variant of image-based

3D reconstruction. The Kinect sensor is a horizontal bar connected to a small base with a

motorized pivot and is designed to be positioned lengthwise above or below the video

display. The device features an "RGB camera, depth sensor and multi-array microphone

running proprietary software",which provide full-body 3D motion capture, facial recognition

and voice recognition capabilities. The depth sensor consists of an infraredlaser projector

combined with a monochrome CMOS sensor, which captures video data in 3D under any

ambient light conditions. The sensing range of the depth sensor is adjustable, and the Kinect

software is capable of automatically calibrating the sensor based on the physical

environment, accommodating for the presence of the user.

Figure 4.1

23

Figure 4.2

Described by Microsoft personnel as the primary innovation of Kinect, the software

technology enables advanced gesture recognition, facial recognition and voice recognition.

Kinect sensor outputs video at a frame rate of 30 Hz. The RGB video stream uses 8-bit VGA

resolution (640 × 480 pixels) with a Bayer color filter, while the monochrome depth sensing

video stream is in VGA resolution (640 × 480 pixels) with 11-bit depth, which provides

2,048 levels of sensitivity. The Kinect sensor has a practical ranging limit of 1.2–3.5 m (3.9–

11 ft) distance when used with the Xbox software. The area required is roughly 6m², although

the sensor can maintain tracking through an extended range of approximately 0.7–6 m (2.3–

20 ft). The sensor has an angular field of view of 57° horizontally and 43° vertically, while

the motorized pivot is capable of tilting the sensor up to 27° either up or down. The

horizontal field of the Kinect sensor at the minimum viewing distance of ~0.8 m (2.6 ft) is

therefore ~87 cm (34 in), and the vertical field is ~63 cm (25 in), resulting in a resolution of

just over 1.3 mm (0.051 in) per pixel. Because the Kinect sensor's motorized tilt mechanism

requires more power than the Xbox 360's USB ports can supply, the device makes use of a

proprietary connector combining USB communication with additional power.

The kinect sensor has been designed to work with the XBOX 360 gaming station, so we take

the help of third party developers and open source drivers which make it compatible for use

with a computer.

24

Microsoft released Kinect software development kit for windows 7 in june 2011 which

allows developers to write Kinecting apps in C++/CLI, C#, or Visual Basic .NET. Thus, it

involves immense scope of research and development being fairly new leap in the field of

depth image processing. It includes the following features:

1. Raw sensor streams: Access to low-level streams from the depth sensor, color camera

sensor, and four-element microphone array.

2. Skeletal tracking: The capability to track the skeleton image of one or two people

moving within the Kinect field of view for gesture-driven applications.

3. Advanced audio capabilities: Audio processing capabilities include sophisticated

acoustic noise suppression and echo cancellation, beam formation to identify the

current sound source, and integration with the windows speech recognition API.

4. Sample code and documentation.

4.1.1 Using Kinect

Hardware connections:

The Kinect is provided with a normal USB connector and split to a normal power adapter.

Using it over a prolonged period of time gets the sensor heated up so regular break intervals

of one hour should be taken so as to avoid any damage.

4.1.2 Kinect Drivers

There are three options to get the driver working on Windows platform:

CLNUI (untested).

Open-Kinect.

OpenNI/NITE.

25

Here we are using OpenNI/NITE drivers and this includes the following software:

OpenNI 1.0.0.25 Unstable.

Primesense NITE 1.3.0.18 Unstable.

SensorKinect 0.4.

Microsoft Visual C++ Redistributable.

Visual Studio 2010.

Although there are a variety of versions of the above software, we are using the above

mentioned ones because these are compatible to be used with MATLAB. For using it, the

function files have to be linked with mex files which are based on Visual Studio. From the

file exchange database of MATLAB Central, some C++ wrapper functions for Kinect and

OpenNI library were downloaded and linked into MATLAB by compiling the cpp files.

The main aim of using Kinect is to track a user and detect the movements so that the gesture

can be impersonated via a humanoid. For this purpose, skeletal tracking of the human body is

done. At the heart of Kinect, lies a time of flight camera that measures the distance of any

given point from the sensor using the time taken by near-IR light to reflect from the object. In

addition to it, an IR grid is projected across the scene to obtain deformation information of

the grid to model surface curvature. Cues from RGB and depth stream from the sensor are

used to fit a stick skeleton model to the human body which has been described later.

4.2 ATMEL AVR:

The AVR is a modified Harvard architecture8-bitRISC single chip microcontroller

which was developed by Atmel in 1996. The AVR was one of the first microcontroller

families to use on-chip flash memory for program storage, as opposed to one-time

programmable ROM, EPROM, or EEPROM used by other microcontrollers at the time.

Flash, EEPROM, and SRAM are all integrated onto a single chip, removing the need for

external memory in most applications. Some devices have a parallel external bus option to

allow adding additional data memory or memory-mapped devices. Almost all devices (except

26

the smallest TinyAVR chips) have serial interfaces, which can be used to connect larger

serial EEPROMs or flash chips.

Current AVRs offer a wide range of features:

Multifunction, bi-directional general-purpose I/O ports with configurable, built-in

pull-up resistors

Internal, self-programmable instruction flash memory up to 256 kB (384 kB on

XMega)

In-system programmable using serial/parallel low-voltage interfaces

debugWIRE uses the /RESET pin as a bi-directional communication channel

to access on-chip debug circuitry. It is present on devices with lower pin

counts, as it only requires one pin.

Internal data EEPROM up to 4 kB

Internal SRAM up to 16 kB

Analog comparator

10 or 12-bit A/D converters, with multiplex of up to 16 channels

12-bit D/A converters

A variety of serial interfaces, including

Synchronous/asynchronous serial peripherals (UART/USART) (used with RS-

232, RS-485, and more)

Serial Peripheral Interface Bus (SPI)

Universal Serial Interface (USI) for two or three-wire synchronous data

transfer

Multiple power-saving sleep modes

Lighting and motor control (PWM-specific) controller models

USB controller support

Ethernet controller support

LCD controller support

27

4.2.1 Atmega 16

The high-performance, low-power Atmel 8-bit AVR RISC-based microcontroller

combines 16KB of programmable flash memory, 1KB SRAM, 512B EEPROM, an 8-channel

10-bit A/D converter, and a JTAG interface for on-chip debugging. The device supports

throughput of 16 MIPS at 16 MHz and operates between 4.5-5.5 volts.

By executing instructions in a single clock cycle, the device achieves throughputs

approaching 1 MIPS per MHz, balancing power consumption and processing speed.

PIN DIAGRAM

Figure 4.3

28

4.2.2 Pin Descriptions

VCC - Digital supply voltage.

GND - Ground.

Port A (PA7..PA0) - Port A serves as the analog inputs to the A/D Converter. Port A also

serves as an 8-bit bi-directional I/O port, if the A/D Converter is not used.

Port B (PB7..PB0) - Port B is an 8-bit bi-directional I/O port with internal pull-up resistors

(selected for each bit). The Port B output buffers have symmetrical drive characteristics with

both high sink and source capability.

Port C (PC7..PC0) - Port C is an 8-bit bi-directional I/O port with internal pull-up resistors

(selected for each bit). The Port C output buffers have symmetrical drive characteristics with

both high sink and source capability.

Port D (PD7..PD0) - Port D is an 8-bit bi-directional I/O port with internal pull-up resistors

(selected for each bit). The Port D output buffers have symmetrical drive characteristics with

both high sink and source capability.

RESET - Reset Input. A low level on this pin for longer than the minimum pulse length will

generate a reset, even if the clock is not running.

XTAL1 - Input to the inverting Oscillator amplifier and input to the internal clock operating

circuit.

XTAL2 - Output from the inverting Oscillator amplifier.

AVCC - AVCC is the supply voltage pin for Port A and the A/D Converter. It should be

externally connected to VCC, even if the ADC is not used. If the ADC is used, it should be

connected to VCC through a low-pass filter.

AREF - AREF is the analog reference pin for the A/D Converter.

4.2.3 Using the USART of ATMEGA16

A Universal Asynchronous Receiver/Transmitter, abbreviated UART is a type of

"asynchronous receiver/transmitter", a piece of computer hardware that translates data

between parallel and serial forms. UARTs are commonly used in conjunction with

communication standards such as EIA, RS-232, RS-422 or RS-485. The universal

designation indicates that the data format and transmission speeds are configurable and that

29

the actual electric signaling levels and methods (such as differential signaling etc.) typically

are handled by a special driver circuit external to the UART.

A UART is usually an individual (or part of an) integrated circuit used for serial

communications over a computer or peripheral device serial port. UARTs are now commonly

included in microcontrollers. A dual UART, or DUART, combines two UARTs into a single

chip. Many modern ICs now come with a UART that can also communicate synchronously;

these devices are called USARTs (universal synchronous/asynchronous receiver/transmitter).

Like many microcontrollers AVR also have a dedicated hardware for serial communication

this part is called the USART - Universal Synchronous Asynchronous Receiver Transmitter.

This special hardware makes your life as programmer easier. You just have to supply the data

you need to transmit and it will do the rest. As you saw serial communication occurs at

standard speeds of 9600, 19200 bps etc and this speeds are slow compared to the AVR CPUs

speed. The advantage of hardware USART is that you just need to write the data to one of the

registers of USART and your done, you are free to do other things while USART is

transmitting the byte.

Also the USART automatically senses the start of transmission of RX line and then inputs the

whole byte and when it has the byte it informs you (CPU) to read that data from one of its

registers.

The USART of AVR is very versatile and can be setup for various different modes as

required by your application. In this tutorial I will show you how to configure the USART in

a most common configuration and simply send and receive data. Later on I will give you my

library of USART that can further ease you work. It will be little complicated (but more

useful) as it will have a FIFO buffer and will use interrupt to buffer incoming data so that you

are free to anything in your main () code and read the data only when you need. All data is

stored into a nice FIFO (first in first out queue) in the RAM by the ISR.

The USART of the AVR is connected to the CPU by the following six registers.

UDR - USART Data Register: Actually this is not one but two register but when we

read it we will get the data stored in receive buffer and when we write data to it goes

into the transmitter‟s buffer.

UCSRA - USART Control and status Register A : As the name suggests it is used to

configure the USART and it also stores some status about the USART. There are two

more of these kinds the UCSRB and UCSRC.

UBRRH and UBRRH : This is the USART Baud rate register, it is 16BIT wide so

UBRRH is the High Byte and UBRRL is Low byte. But as we are using C language it

is directly available as UBRR and compiler manages the 16BIT access.

So the connection of AVR and its internal USART can be visualized as follows.

Figure 4.4

30

UDR : USART Data Register

UCSRA : USART Control and Status Register A

UCSRB : USART Control and Status Register B

UCSRC : USART Control And Status Register C

UBRR : USART Baud Rate Register:UBRRH is the High Byte and UBRRL is Low

byte.

4.3 ZIGBEE Communication:

ZigBee is a standard that defines a set of communication protocols for low-data-rate, very

low power, short-range wireless networking. ZigBee is a specification for a suite of high level

communication protocols using small, low-power digital radios based on an IEEE 802

standard for personal area networks. ZigBee-compliant radios may operate on one of three

different radio bands: the 800 MHz, 900 MHz, or 2.4 GHz frequencies. The technology

defined by the ZigBee specification is intended to be simpler and less expensive than other

WPANs, such as Bluetooth. ZigBee is targeted at radio-frequency (RF) applications that

require a low data rate, long battery life, and secure networking.

4.3.1 XBEE

This is the hardware that we are using which supports ZIGBEE communication.

The XBee and XBee-PRO OEM RF Modules were engineered to meet IEEE 802.15.4

standards and support the unique needs of low-cost, low-power wireless sensor

networks.

Figure 4.5

31

The modules require minimal power and provide reliable delivery of data between

devices.

The modules operate within the ISM 2.4 GHz frequency band and are pin-for-pin

compatible with each other.

4.3.2 XBEE Working Principle

The XBEE RF modules interface to a host device through a logic-level asynchronous

serial port. Through its serial port, the module can communicate with any logic and

voltage compatible UART.

Data enters the module UART through pin 3 as an asynchronous serial signal. The

signal should be idle high when no data is being transmitted.

Each data byte has a start bit (low), 8 data bits (least significant bit first) and a stop bit

(high).

Figure 4.6

32

CHAPTER 5

PROCESS INVOLVED

The dynamic process of behavioural synchronization through a humanoid begins

with derivation of real time information about the joints of the user be it prismatic (linear) or

revolute (rotational). With degree of freedom as high as 18 the task becomes extremely

tedious and requires tremendous processing power. One of the major problems in monitoring

multiple joint information is the definition of the reference frame with respect to which

information about the other frames can be derived.

33

Even though the reference for individual joints may vary it is convenient to process all

the joints with a single reference point. The Kinect Camera provides this information to the

central control block using a combination of infrared imaging and RGB camera. The infrared

laser creates a depth profile of the area in display and the RGB camera returns RGB matrix.

Since the Kinect sensor is specifically designed for gaming purposes it intrinsically calculates

the distance between the user and itself with a defined reference frame. The reference frame

for the Kinect sensor is a predefined location approximately at the center of the view. The

horizontal right hand side of observation denotes the x axis and transverseaxisdenotes y axis.

The z axis is the linear distance between the Kinect and user, the farther the user the higher

the z value.

5.1 Data Points from KINECT:

Using the Kinect we monitor the real time positions and orientation of 15 different data

points in the user‟s body, namely:

Head

Neck

Left Shoulder

Left elbow

Left hand

Right Shoulder

Right elbow

Right hand

Torso

Left Hip

Left Knee

Left Foot

Right Hip

Right knee

Right foot

34

The sensor generates a 2D matrix of 225 x 7 dimension with the above mentioned

data points for a single person anda maximum tracking capability of 15 people. The seven

columns are User ID, Tracking Confidence, x, y,z (coordinates in mm) and the next X,Y (the

pixel value of the corresponding data point). Using this information we calculate:

The joint angles in different planes between different links.

The link lengths between data points.

5.1.1 Calculation of Joint Angles

Having raw joints is not enough for further operations, because of that on the next

step the vector between joints should be constructed.As an example below a description of

construction of vector between joints of right hand has been given:

Connecting the right elbow, right hand and right elbow and right shoulder determines

the angle for right elbow.

Connecting the right shoulder, right elbow and right shoulder and neck determines the

angle for right shoulder.

Now the angle between the vectors is calculated by using a simple geometrical

operation such as angle between two vectors i.e.

𝑢. 𝑣 = cos(𝜃) 𝑢 𝑣

5.1.2 Calculation of Link Lengths

Using the x, y and z coordinates derived from Kinect sensor and using the formula for

distance calculation the link lengths are found.

𝑙𝑒𝑛𝑔𝑡ℎ = (𝑥2 − 𝑥1)2 + (𝑦2 − 𝑦1)2 + (𝑧2 − 𝑧1)2

35

After calculating the joint angles and link lengths, the next step in the behavioral

synchronization would be to realize these real time parameters in the desired form. This step

includes conversion of the parameters to actual servo angles. This step is useful for tuning

purpose between the real angle values of joints and robot servo values. Another reason for the

necessity of this step is because the real human joint angle values range from 0~360°, when

the robot servo motor can accept values from 0~255.

Now that the destination coordinates of the individual tool frames for the humanoid

robot have been derived, two simultaneous steps follow, embedding the gesture movements

into the virtual (software) and real (hardware) model.

5.2 Denavit-Hartenberg Methodology:

The virtual model is designed based on principles of kinematic modeling by Denavit-

Hartenberg methodology for robotic manipulators. There are total 18 DOF (Degrees of

Freedom). Each arm has 5 DOF and each leg has 4 DOF.The destination coordinates arefed

into the ADAMS plant which has been configured to take input in the form of angular

velocities and output the current joint angles. The ADAMS plant is executed in MATLAB

(Simulink) environment with a function file to control it.There are two separate methods to

control the motion of a humanoid i.e. DC (Direct Control) and CC (Command

Control).Direct Control performs the conversion of 3D joint coordinates of the user to servo

angles whereas the Command Control converts information to a single command for the

robot‟s understanding. The information that is sent is the destination positions for the tool

frame and the calculation of joint angles is left for the model.

Since the DOF for each arm is 3, the tool frame coordinates – x, y and z depends on 3

angles 𝜃1, 𝜃2𝑎𝑛𝑑𝜃3 which can be expressed as:

𝑥𝑦𝑧 = 𝑇

𝜃1

𝜃2

𝜃3

Here „T‟ is the transformation matrix that relates the coordinates with the joint angles.

36

𝑥𝑦 𝑧

= 𝐽

𝜃1

𝜃2

𝜃3

„J‟ denotes the jacobian matrix and 𝑥 , 𝑦 𝑎𝑛𝑑𝑧 are the product of error signal of the

corresponding axis and the proportionality constant𝑘𝑝 and𝜃1 , 𝜃2

𝑎𝑛𝑑𝜃3 are the joint anglular

velocities.

𝑥 = (𝑥𝑑𝑒𝑠𝑡𝑖𝑛𝑎𝑡𝑖𝑜𝑛 − 𝑥) × 𝑘𝑝

𝑦 = (𝑦𝑑𝑒𝑠𝑡𝑖𝑛𝑎𝑡𝑖𝑜𝑛 − 𝑦) × 𝑘𝑝

𝑧 = (𝑧𝑑𝑒𝑠𝑡𝑖𝑛𝑎𝑡𝑖𝑜𝑛 − 𝑧) × 𝑘𝑝

To evaluate the angular velocities, we use the expression,

𝜃1

𝜃2

𝜃3

= 𝐽−1 𝑥𝑦 𝑧

37

5.2.1 Simulink Model

Figure 5.1

38

Moving on to the hardware model by analyzing the simulation output, it was observed

that the command control strategy was not ideal for hardware based platform as the solution

space for the simulation model may not be similar with the model. So in order to avoid

singularity points in the movement, direct control strategy is implemented. For the same, the

Kinect sensor is configured not to calculate the destination for the tool frame but the

individual joint angles that are required for behavioral synchronization. After the joint angles

have been derived, there comes a need to communicate these angles in a suitable format for

the microcontrollers to recognize and implement. A wireless communication technique was

employed according to the ZIGBEE standard protocol via XBEE trans-receiver modules.

5.3 Modes of Operation of ZIGBEE:

As mentioned earlier, the technology defined by the ZigBee specification is intended

to be simpler and less expensive than other WPANs, such as Bluetooth. ZigBee is targeted at

radio-frequency (RF) applications that require a low data rate, long battery life, and secure

networking.

One of the Xbee pair is connected to the PC using Xbee USB adapter board and the

other one is interfaced through the development board using a microcontroller. The module

can communicate with any logic and voltage compatible UART through a logic –level

asynchronous serial port. The data enters the module‟s UART from the serial port of the PC

through the DI pin as an synchronous serial signal. The signal should idle be high when no

data is being transmitted. Each data byte consists of a start bit(low), 8 databits (least

significant being first) and a stop bit.

There are two modes of operation:

1.Transparent mode

39

In this mode all UART data received through the DI pin is queued up for RF

transmission. Data is buffered in the DI buffer until the data is packetized and transmitted.

When RF data is received the data is sent out through the DO pin.

2. API (Application Programming Interface)

In API mode all the data entering and leaving the module is contained in frames that

defines the operations or events within the module.

The default operation which is used by Xbee is transparent mode. The combination of

XCTU software and Xbee adapter is used to configure the baud rate, channel number and

network ID of Xbeemodule .The advantage of using Zigbee is that it consumes less power

than WiFi and has longer range than Bluetooth. It operates in the Industrial, Scientific and

Medical (ISM) thus making it the best option for robot communication.

5.4 Memory Addressing Technique:

Now that the information to be sent and method of communication is ready, the next

step is finding out the memory addressing technique so that the data transmitted should be in

synchronization with the data received. In order to address the 18 actuated joints with a single

duplex channel of 8 bit word length we had to involve an addressing strategy. The number of

bits required to address 18 individual motors would ideally be 5 bits and to transmit 8 bits of

data to each of the motors makes the word length of 13 bits which is not available. So instead

of delivering the word in a single transmission the information is broken in 2 nibbles of 4 bit

length and 2 consecutive transmissions are required to successfully transmit the 8 bit of data.

With this split we had to assign 1 bit to distinguish between the higher and the lower nibble

which again created a problem as the word length of a transmission is limited to 8 bits only.

A new way of addressing had to be involved to understand it we imagine a room with 2 doors

each with a different key to access it. Only when you have successfully unlocked both the

locks with 2 separate keys you can access the room. Metaphorically the motors are the room

and the keys are the address to unlock it. 2 separate 4 bit address for each motor are used to

send the higher and lower nibble of data from the source. This scheme allows us to address a

40

maximum of 256 different addressees and transmit information of 8 bits to each one in 2

consecutive transmission.

5.5 Controlling the Servo Motor:

After the reception of the control signal at the remotely located manipulators the

microcontrollers controlling the servos decodes the servo angles from the control signal and

implements it. Controlling a servo using a microcontroller requires no external driver like H-

bridge only a control signal needs to be generated for the servo to position it in any fixed

angle. The standard frequency for control signal is 50Hz and the duty cycle controls the

angle.

Figure 5.2

The timing for standard servos for efficient control of the servo angle is as follows:

0.388ms = 0 degree.

1.264ms = 90 degrees. (Neutral position)

2.14ms = 180 degrees.

The control signal is 50Hz (i.e. the period is 20ms) and the width of

positive pulse controls the angle. This implementation of the servo angle

completes 1 cycle of the process as it ends with the beginning of the kinect

sensor inputting the joint information to the control system.

41

CHAPTER 6

CODE IMPLEMENTED

Coding takes up a major role in deciding the operation of any work process. Here,

the coding takes place in various stages, namely:

For KINECT (in MATLAB).

For ADAMS.

For the Microcontrollers implanted in Humanoid.

All the codes along with the outputs have been illustrated in the following pages.

42

6.1 Coding in KINECT:

From the functional coding in MATLAB, the following parameters were extracted:

RGB Image.

IR Image.

Skeleton Image of the User.

6.1.1 RGB Image.

Code – addpath('Mex')

SAMPLE_XML_PATH='Config/SamplesConfig.x

ml';

KinectHandles=mxNiCreateContext(SAMPLE_X

ML_PATH);

figure;

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2

1]);

subplot(1,2,1),h1=imshow(I);

subplot(1,2,2),h2=imshow(I,[0 9000]);

colormap('jet');

for i=1:90

Figure 6.1

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

D=mxNiDepth(KinectHandles); D=permute(D,[2 1]);

set(h1,'CDATA',I);

set(h2,'CDATA',D);

drawnow;

end

% Stop the Kinect Process

mxNiDeleteContext(KinectHandles);

43

6.1.2 IR Image.

Code – addpath('Mex')

SAMPLE_XML_PATH='Config/SampleIRConfig.xml';

% Start the Kinect Process

KinectHandles=mxNiCreateContext(SAMPLE_XML_PATH);

figure;

J=mxNiInfrared(KinectHandles); J=permute(J,[2 1]);

h=imshow(J,[0 1024]);

for i=1:9000

J=mxNiInfrared(KinectHandles); J=permute(J,[2 1]);

set(h,'Cdata',J);

end

% Stop the Kinect Process

mxNiDeleteContext(KinectHandles);

Figure 6.2

44

6.1.3 Skeleton Image.

Code –

addpath('Mex')

SAMPLE_XML_PATH='Config/SamplesConfig.xml';

KinectHandle=mxNiCreateContext(SAMPLE_XML_PATH);

while(Pos(1)==0);

mxNiUpdateContext(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

Pos= mxNiSkeleton(KinectHandles);

set(h,'Cdata',I); drawnow;

end

hh=zeros(1,9);

while(Pos(1)>0)

mxNiUpdateContext(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

set(h,'Cdata',I); drawnow;

if(hh(1)>0);

for i=1:9, delete(hh(i)); end

end

hold on

y=Pos(1:15,7);

x=Pos(1:15,6);

hh(1)=plot(x,y,'r.');

hh(2)=plot(x([13 14 15]),y([13 14 15]),'g');

hh(3)=plot(x([10 11 12]),y([10 11 12]),'g');

hh(4)=plot(x([9 10]),y([9 10]),'m');

hh(5)=plot(x([9 13]),y([9 13]),'m');

hh(6)=plot(x([2 3 4 5]),y([2 3 4 5]),'b');

hh(7)=plot(x([2 6 7 8]),y([2 6 7 8]),'b');

hh(8)=plot(x([1 2]),y([1 2]),'c');

hh(9)=plot(x([2 9]),y([2 9]),'c');

drawnow

end

mxNiDeleteContext(KinectHandles);

45

Figure 6.3

Using the skeletal image shown above, two parameters were found:

Link lengths.

Angle between required joints.

6.1.4 Finding out the link lengths.

Code –

addpath('Mex')

SAMPLE_XML_PATH='Config/SamplesConfig.xml';

KinectHandles=mxNiCreateContext(SAMPLE_XML_PATH);

figure,

Pos= mxNiSkeleton(KinectHandles);

46

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

h=imshow(I);

while(Pos(1)==0);

mxNiUpdateContext(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

Pos= mxNiSkeleton(KinectHandles);

set(h,'Cdata',I); drawnow;

end

hh=zeros(1,9);

while(Pos(1)>0)

mxNiUpdateContext(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

set(h,'Cdata',I); drawnow;

hold on

y=Pos(1:15,7);

x=Pos(1:15,6);

sx= Pos(6,3);

sy= Pos(6,4);

sz= Pos(6,5);

ex= Pos(7,3);

ey= Pos(7,4);

ez= Pos(7,5);

hx = Pos(8,3);

hy = Pos(8,4);

hz = Pos(8,5);

l1 = (((sz-ez)^2)+((sy-ey)^2)+((sx-ex)^2))^0.5

l2 = (((ez-hz)^2)+((ey-hy)^2)+((ex-hx)^2))^0.5

hh(1)= plot(x,y,'r.');

hh(2)=plot(x([13 14 15]),y([13 14 15]),'g');

hh(3)=plot(x([10 11 12]),y([10 11 12]),'g');

hh(4)=plot(x([9 10]),y([9 10]),'m');

hh(5)=plot(x([9 13]),y([9 13]),'m');

hh(6)=plot(x([2 3 4 5]),y([2 3 4 5]),'b');

hh(7)=plot(x([2 6 7 8]),y([2 6 7 8]),'b');

hh(8)=plot(x([1 2]),y([1 2]),'c');

hh(9)=plot(x([2 9]),y([2 9]),'c');

drawnow

end

mxNiDeleteContext(KinectHandles);

47

Figure 6.4

6.1.5 Finding out the angles.

Code –

addpath('Mex')

SAMPLE_XML_PATH='Config/SamplesConfig.xml';

KinectHandles=mxNiCreateContext(SAMPLE_XML_PATH);

figure,

Pos= mxNiSkeleton(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

h=imshow(I);

while(Pos(1)==0);

mxNiUpdateContext(KinectHandles);

48

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

Pos= mxNiSkeleton(KinectHandles);

set(h,'Cdata',I); drawnow;

end

hh=zeros(1,9);

while(Pos(1)>0)

mxNiUpdateContext(KinectHandles);

I=mxNiPhoto(KinectHandles); I=permute(I,[3 2 1]);

set(h,'Cdata',I); drawnow;

Pos= mxNiSkeleton(KinectHandles,1);

if(hh(1)>0);

for i=1:9, delete(hh(i)); end

end

hold on

y=Pos(1:15,7);

x=Pos(1:15,6);

rshoulder = [ Pos(6,3) Pos(6,4) Pos(6,5) ];

lshoulder = [ Pos(3,3) Pos(3,4) Pos(3,5) ];

relbow = [ Pos(7,3) Pos(7,4) Pos(7,5) ];

rhand = [ Pos(8,3) Pos(8,4) Pos(8,5) ];

rhip = [ Pos(13,3) Pos(13,4) Pos(13,5) ];

torso = [ Pos(9,3) Pos(9,4) Pos(9,5) ];

head = [ Pos(1,3) Pos(1,4) Pos(1,5) ];

rshouldern = [ Pos(6,3) Pos(6,4) ];

lshouldern = [ Pos(3,3) Pos(3,4) ];

relbown = [ Pos(7,3) Pos(7,4) ];

x2 = findslope(rshouldern,lshouldern);

y2 = findslope(rshouldern,relbown);

angle1 = slopeangle(x2,y2);

angle2 = hypot(angle1,0)-76;

torsom = [ Pos(9,4) Pos(9,5) ];

headm = [ Pos(1,4) Pos(1,5) ];

x1 = findslope(torsom,headm);

rshoulderm = [ Pos(6,4) Pos(6,5) ];

relbowm = [ Pos(7,4) Pos(7,5) ];

y1 = findslope(rshoulderm,relbowm);

49

angle = slopeangle(x1,y1);

angle3 = hypot(angle,0)-2;

anc = [ angle2 angle3 ]

hh(1)=plot(x,y,'r.');

hh(2)=plot(x([13 14 15]),y([13 14 15]),'g');

hh(3)=plot(x([10 11 12]),y([10 11 12]),'g');

hh(4)=plot(x([9 10]),y([9 10]),'m');

hh(5)=plot(x([9 13]),y([9 13]),'m');

hh(6)=plot(x([2 3 4 5]),y([2 3 4 5]),'b');

hh(7)=plot(x([2 6 7 8]),y([2 6 7 8]),'b');

hh(8)=plot(x([1 2]),y([1 2]),'c');

hh(9)=plot(x([2 9]),y([2 9]),'c');

drawnow

end

mxNiDeleteContext(KinectHandles);

Figure 6.5

50

6.2 Coding in ADAMS - MATLAB:

The complete virtual model was embedded with all the multidisciplinary values and the plant

was exported to MATLAB. This links the model between ADAMS and MATLAB so that

after executing this code, the Simulink model is generated, the execution of which drives the

virtual model so that the desired results can be obtained.

After simulating the model, the torque profiles for the joints of right arm were extracted:

Figure 6.6

Figure 6.7 Figure 6.8

51

It can be observed from Fig that the torque required is at the maximum position i.e. the crest

of the graph. From Fig , it can be seen that the torque profile changes according to the

position of the hand and Fig represents the elbow joint. It was programmed to have a

continuous to-and-fro motion and due to this the graph is obtained as a sine wave with

increasing amplitude.

After that, input was taken in from the KINECT camera and it was fed into the model so that

there is synchronization between the user and the humanoid.

Figure 6.9

52

6.3 Coding in the Microcontrollers:

For driving two servo motors one Atmega microcontroller, one BEC (Battery Eliminator

Circuit) is used. For the different microcontrollers, different address has to be assigned which

was explained in the Process Involved section. The basic difference between the coding of

different microcontrollers is just the addressing value. Shown below is the code which is

implemented:

#include<avr/io.h>

#include<util/delay.h>

#include<xslcd.h>

#include <avr/interrupt.h>

voidUSARTInit(uint16_t); // Function declarations.

charUSARTReadChar(void);

voidUSARTWriteChar(char);

voidUSARTInit(uint16_t ubrr_value) // Function for initializing USART.

{

UBRRL = ubrr_value;

UBRRH = (ubrr_value>>8);

UCSRC=(1<<URSEL)|(3<<UCSZ0);

UCSRB=(1<<RXEN)|(1<<TXEN);

}

charUSARTReadChar(void) // Function for Reading Value from USART.

{

while(!(UCSRA & (1<<RXC)));

return UDR;

}

int main(void) // Main function starts.

{

unsigned char data;

unsigned char add1,add2;

unsigned char vl,vh,value;

TCCR1A|=(1<<COM1A1)|(1<<COM1B1)|(1<<WGM11);

TCCR1B|=(1<<WGM13)|(1<<WGM12)|(1<<CS11)|(1<<CS10);

ICR1=2499; // Registers are set with values for initializing timers.

value = 0x00;

InitLCD(); // LCD and USART are initialized.

USARTInit(51);

LCDClear();

53

LCDWriteStringXY(0,0,"Initialising...");

_delay_ms(160);

LCDClear();

LCDWriteStringXY(0,0,"M1=");

LCDWriteStringXY(0,1,"M2=");

DDRD|=(1<<PD4)|(1<<PD5);

while(1)

{

data = USARTReadChar(); // Value is read from USART.

_delay_ms(100);

add1 = data & 0xf0;

switch (add1) // The higher nibble is compared to first 4 bits.

{

case 0x00:

vl = data & 0x0f;

data = USARTReadChar(); // If it is satisfied, next value is read from USART.

add2 = data & 0xf0;

if (add2 == 0x80) // After comparing it to the next value, if the

{ // conditions are satisfied, the information about angles

vh = data<<4; // is extracted and fed to the motor.

value = vl | vh;

if (value < 180)

{

OCR1B = 58+(1.55*value);

LCDWriteIntXY(4,0,value,3);

}

else

LCDWriteStringXY(4,0,'Invalid Angle');

break;

}

else

break;

break;

case 0x10:

vl = data & 0x0f;

data = USARTReadChar();

add2 = data & 0xf0;

if (add2 == 0x90)

{

vh = data<<4;

value = vl | vh;

if (value < 180)

{

OCR1A = 56+(1.55*value);

LCDWriteIntXY(4,1,value,3);

54

}

else

LCDWriteStringXY(4,1,'invalid angle');

break;

}

else

break;

break;

default:

add1 = 0xff;

}

}

return 1;

}

This is the code for programming one of the MCU board. Based upon the address assigned to

the MCU, the code changes.

55

CHAPTER 7

CONCLUSION

The implemented architecture suggests simple, low cost and robust solution for

controlling humanoid with human gestures. The complete system setup requires the

synchronization of software and hardware. Some problems were faced in the field of setting

up the hardware i.e. motors and actual structure.Thus for demonstration of control via human

gestures, the upper-body was mounted on a mobile platform with four wheels with

differential drive system to maneuver. For the future work, the implementation of humanoid

will be expanded to a bipedal system for control based onvarious concepts such as Zero

Moment Point (ZMP) concept, Force Rotation Indicator (FRI), Support Polygon, etc.

56

BIBLIOGRAPHY

[1] Vitor M. F. Santos, Filipe M. T. Silva, “Engineering Solutions To Build An

Inexpensive Humanoid Robot Based On A Distributed Control

Architecture”.2005 5th Ieee-Ras International Conference On Humanoid

Robots.

[2] Milton Ruas, Filipe M. T. Silva, Vitor M. F. Santos. “A Low-Level Control

Architecture For A Humanoid Robot”.

[3] Elvedinkljuno, Robert L.Williams Ii. . “Humanoid Walking Robot: Modeling,

Inverse Dynamics, And Gain Scheduling Control”. Hindawi Publishing

Corporation Journal Of Robotics.

[4] James Kuffner, Koichi Nishiwaki, Satoshi Kagami, Masayuki Inaba,

Hirochika Inoue. “Motion Planning For Humanoid Robots”. 11th Int‟l Symp.

Of Robotics Research (Isrr 2003)

[5] Stefan Waldherr, Roseli Romero, Sebastian Thrun. “A Gesture Based

Interface For Human-Robot Interaction”. 2000 Kluwer Academic Publishers.

[6] David Kortenkamp, Eric Hubber, R. Peter Bonasso. “Recognising and

interpreting gestures on mobile Robot”. Robotics and Automation Group,

NASA.

[7] Mark Becker, EfthimiaKefalea, EricMael,Christop Von DerMalsburg,

MikePagel, J Triesch,Jan C. VorbrUggen, Rolf P. Wurtz,

StefanZadel.GripSee: “A Gesture-controlled Robot forObject Perception and

Manipulation”

[8] Christian Plagemann, VarunGanapati, Daphne Koller, Sebastian Thrun. “Real-

time identification and Localization of Body parts from Depth Images” 2010

www.hindawi.com

[9] Luciano Spinello, Kai O. Arras, Rudolph Triebel, Roland Siegwart. “A

layered Approach to people detection in 3D Range Data”. 2010 www.aaai.org