2D Hand Gesture Recognition - cnx.org

2D Hand Gesture Recognition

By:Tracy Fu

Zihe Huang

2D Hand Gesture Recognition

By:Tracy Fu

Zihe Huang

Online:< http://cnx.org/content/col11606/1.1/ >

C O N N E X I O N S

Rice University, Houston, Texas

This selection and arrangement of content as a collection is copyrighted by Tracy Fu, Zihe Huang. It is licensed under

the Creative Commons Attribution License 3.0 (http://creativecommons.org/licenses/by/3.0/).

Collection structure revised: December 19, 2013

PDF generated: December 19, 2013

For copyright and attribution information for the modules contained in this collection, see p. 40.

Table of Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Matlab Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Our Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Poster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37Attributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

iv

Available for free at Connexions <http://cnx.org/content/col11606/1.1>

Chapter 1

Introduction1

1.1 Introduction and Background

Hand gesture is useful for delivering information that cannot be conveyed through speech or type fromhumans. In this project, we developed a 2D hand gesture recognition method which is able to identifysimple static hand gestures like �stop�, �point�, �okay� and �up� etc. Hand gesture recognition can be appliedto the area of security, remote control, and assistance of certain groups of disabled people(e.g. translatingsign language). Video games industry is an early adopter of this technology because it provides a moreintuitive way for humans to interact with computers.

1This content is available online at <http://cnx.org/content/m48486/1.1/>.


1

2 CHAPTER 1. INTRODUCTION


Chapter 2

Database1

2.1 Database

Our team applies seven categories of hand gestures. They are, namely, Okay, Stop, Up, Down, Point, Metal,Vulcan. For each category, we collect ten pictures from ten di�erent people and treat these 70 pictures inall as the training set. Here we show one of them from each category.



3

4 CHAPTER 2. DATABASE

Okay

Figure 2.1: "Okay" Gesture


5

Stop

Figure 2.2: "Stop" Gesture



Up

Figure 2.3: "Up" Gesture


7

Down

Figure 2.4: "Down" Gesture



Point

Figure 2.5: "Point" Gesture


9

Metal

Figure 2.6: "Metal" Gesture



Vulcan

Figure 2.7: "Vulcan" Gesture


Chapter 3

Feature Extraction1

We start with Vulcan! Here is an original picture for our gesture.



11

12 CHAPTER 3. FEATURE EXTRACTION

Vulcan

Figure 3.1: The Original "Vulcan" Gesture


13

Convert image to grayscale.



Grayscale

Figure 3.2: Vulcan converted to grayscale


15

Filter out noise and convert the image into a binary image.



Binary

Figure 3.3: Binary Form of Vulcan


17

Label the largest connected component in our image matrix and get the edge of our gesture.



Edge

Figure 3.4: Edge of Vulcan


19

Get polar form of the edge matrix. Divide the polar form into n(here n=8) equal bins and count numberof 1's(dots) in each bin.



Demo

Figure 3.5: A demonstration of how our algorithm works


21

Finally, we compare the data we get here with distribution of dots in our training set and get our predictedgesture of this image


Chapter 4

Matlab Code1

Driver.m

clear all

close all

I1 = getnewvector('metal/metal3.JPG',800, 700, 1400, 800);



I4=getnewvector('down1.JPG',800, 700, 1400, 800);



% need to rescale all images here

[x, y,I1]=getcenter(I1);

[x1, y1,I2]=getcenter(I2);



[x4,y4,I5]=getcenter(I5);

[x5,y5,I6]=getcenter(I6);

[theta1, r1] = polarize(I1, [x,y]);

[theta2, r2] = polarize(I2, [x1,y1]);





interval=pi/18;

[ counter1, dividedData1 ] = divideIntoBins(theta1, r1, interval);






counter2 = counter2/(length(theta2));







23

24 CHAPTER 4. MATLAB CODE

% figure;

% subplot(3,1,1);

% plot(counter1);

% subplot(3,1,2);

% plot(counter2);

% subplot(3,1,3);

% plot(counter3);

disp(norm(counter1-counter2,2));




%n1=(norm(counter2,1)+norm(counter3,1))/2

%n2=(norm(counter4,1)+norm(counter5,1)+norm(counter6,1))/3

%

% % comparing within the same category

% v1 = getnewvector('vulcan1.JPG');



% d = zeros(3,1);

% inner = zeros(3,1);

% [d(1),inner(1)] = getdistance(v1,v2);



% %

getnewvector.m

function I=getnewvector(filename, XMIN, YMIN, WIDTH, HEIGHT)

I=imread(filename);

I=imcrop(I,[XMIN, YMIN, WIDTH, HEIGHT]); %crop

I=rgb2gray(I);

filter12=fspecial('gaussian',[20,20],20); %lowpass filter

I=imfilter(I,filter12);

% figure;

% imshow(I);

%I=imcontour(I,1);

% magical contour algorithm %get rid of unconnected dots and get the largest connect part which is our hand

I=mean(double(I)/255,3)>0.5;

[L,N] = bwlabel(I,4);

s=zeros(1,N);

for i=1:N,

s(i)=sum(L(:)==i);

end;

[t,ind]=max(s);

I=double(L==ind);

% figure;

% imshow(I);


25

% filter12=fspecial('gaussian',[17,17],17);

% I=imfilter(I,filter12);

figure;

imshow(I);

% I = imresize(I,[1601 1601]);

% I = im2bw(I,.6);

end

getcenter.m

% get the center of the contour image

% output the center coordinate, and the edged image

function [x,y,Iout]=getcenter(I)

[m, n]=size(I);

a=sum(sum(I));

rowindex=zeros(a,1);

colomnindex=zeros(a,1);

number1=0;

for i=1:m,

for j=1:n,

if I(i,j)==1

number1=number1+1;

rowindex(number1,1)=i;

colomnindex(number1,1)=j;

end

end

end

x=mean(rowindex);

size(x);

x=round(x);

y=mean(colomnindex);

y=round(y);

Iout = edge(I,'canny');

end

processCropped.m

function [ x1,y1 ] = processCropped( category,fileName,saveName)

% automate the process of feature extraction

% Detailed explanation goes here

dir = strcat(category,'/',fileName);

I = getnewvector(dir,0, 0, 1601, 1601);

[x1,y1,I] = getcenter(I);

dir2 = strcat(category,'/',saveName); % save directory

imwrite(I,dir2,'bmp');

end

processCategory.m



function processCategory( category, numOfPics, saveAbrev )

% process the entire category

% Detailed explanation goes here

center = zeros(numOfPics,2);

for n = 1:numOfPics,

fileName = strcat(category,num2str(n),'.jpg');

saveName = strcat(saveAbrev,num2str(n),'.bmp');

center(n,:) = processCropped( category,fileName,saveName);

end

csvwrite(strcat(category,'/','centers.dat'),center);

end

polarize.m

function [theta, r] = polarize(binaryMatrix, center)

%POLARIZE Summary of this function goes here

% Center: 1x2

[m,n] = size(binaryMatrix);

theta = zeros(500,1);

r = zeros(500,1);

numOfDots = 0;

for i = 1:m, % for each row

for j = 1:n, % for each column

if (binaryMatrix(i,j) == 1),

numOfDots = numOfDots + 1;

dot = [i,j] - center; % find the relative coordinate with centroid

[theta(numOfDots), r(numOfDots)] = cart2pol(dot(1),dot(2)); % convert into polar

end

end

end

end


27

divideIntoBins.m

Figure 4.1

testDriver.m

clear all

close all

dir =char('down','up','okay','metal','point','stop','vulcan');

right=0;

wrong=0;

for n=1:7,

traning1=csvread(strcat(dir(n,:),'/','features.dat'));

[x y]=size(traning1);

for k=1:x,

training=traning1(k,:);

type=classify(training,2,7,8);

type1=mode(type);



if type1==n

right=right+1;

else

wrong=wrong+1;

end

end

end

size(type)

right

wrong

% I1 = getnewvector('metal/metal3.JPG',800, 700, 1400, 800);



% I4=getnewvector('down1.JPG',800, 700, 1400, 800);



% % need to rescale all images here

%

% [x, y,I1]=getcenter(I1);

% [x1, y1,I2]=getcenter(I2);



% [x4,y4,I5]=getcenter(I5);

% [x5,y5,I6]=getcenter(I6);

% [theta1, r1] = polarize(I1, [x,y]);

% [theta2, r2] = polarize(I2, [x1,y1]);





% interval=pi/18;

% [ counter1, dividedData1 ] = divideIntoBins(theta1, r1, interval);






% counter2 = counter2/(length(theta2));





% % figure;

% % subplot(3,1,1);

% % plot(counter1);

% % subplot(3,1,2);

% % plot(counter2);


29

% % subplot(3,1,3);

% % plot(counter3);

% disp(norm(counter1-counter2,2));




%

% %n1=(norm(counter2,1)+norm(counter3,1))/2

% %n2=(norm(counter4,1)+norm(counter5,1)+norm(counter6,1))/3

%

%

%

% %

% % % comparing within the same category

% % v1 = getnewvector('vulcan1.JPG');



% % d = zeros(3,1);

% % inner = zeros(3,1);

% % [d(1),inner(1)] = getdistance(v1,v2);



% % %



classify.m(part1)

Figure 4.2


31

classify.m(part2)

Figure 4.3


Chapter 5

Our Team1

5.1 The Team

Chenxin(Tracy) FuMinyang MaZihe HuangAoliang ZhiElectrical and Computer Engineering, Rice University, 77005

5.2 Acknowledgement

We would like to thank Professor Rich Baraniuk for his great lectures and Kaushik Mitra for his contributiveadvice.



33

34 CHAPTER 5. OUR TEAM


Chapter 6

Poster1

Group Poster

Figure 6.1: The poster used for presentation



35

36 CHAPTER 6. POSTER


Chapter 7

Results1

Figure 7.1: Number of Correct Recognitions for Each Gesture



37

38 CHAPTER 7. RESULTS

Figure 7.2: Correct Rate for Test Set


39

Figure 7.3: Correct Rate for Training Set


40 ATTRIBUTIONS

Attributions

Collection: 2D Hand Gesture Recognition

Edited by: Tracy Fu, Zihe HuangURL: http://cnx.org/content/col11606/1.1/License: http://creativecommons.org/licenses/by/3.0/

Module: "Introduction"By: Tracy Fu, Zihe HuangURL: http://cnx.org/content/m48486/1.1/Page: 1Copyright: Tracy Fu, Zihe HuangLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Database"By: Tracy Fu, Zihe HuangURL: http://cnx.org/content/m48369/1.1/Pages: 3-10Copyright: Tracy Fu, Zihe HuangLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Feature Extraction"By: Tracy Fu, Zihe HuangURL: http://cnx.org/content/m48474/1.1/Pages: 11-21Copyright: Tracy Fu, Zihe HuangLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Matlab Code"By: Tracy Fu, Zihe HuangURL: http://cnx.org/content/m48483/1.1/Pages: 23-31Copyright: Tracy Fu, Zihe HuangLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Our Team"By: Tracy FuURL: http://cnx.org/content/m48466/1.1/Page: 33Copyright: Tracy FuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Poster"By: Tracy Fu, Zihe HuangURL: http://cnx.org/content/m48380/1.1/Pages: 35-36Copyright: Tracy Fu, Zihe HuangLicense: http://creativecommons.org/licenses/by/3.0/


ATTRIBUTIONS 41

Module: "Results"By: Tracy FuURL: http://cnx.org/content/m48421/1.1/Pages: 37-39Copyright: Tracy FuLicense: http://creativecommons.org/licenses/by/3.0/


About ConnexionsSince 1999, Connexions has been pioneering a global system where anyone can create course materials andmake them fully accessible and easily reusable free of charge. We are a Web-based authoring, teaching andlearning environment open to anyone interested in education, including students, teachers, professors andlifelong learners. We connect ideas and facilitate educational communities.

Connexions's modular, interactive courses are in use worldwide by universities, community colleges, K-12schools, distance learners, and lifelong learners. Connexions materials are in many languages, includingEnglish, Spanish, Chinese, Japanese, Italian, Vietnamese, French, Portuguese, and Thai. Connexions is partof an exciting new information distribution system that allows for Print on Demand Books. Connexionshas partnered with innovative on-demand publisher QOOP to accelerate the delivery of printed coursematerials and textbooks into classrooms worldwide at lower prices than traditional academic publishers.

2D Hand Gesture Recognition - cnx.org

Documents

Transcript of 2D Hand Gesture Recognition - cnx.org