CS231A Course Project Final ReportSign Language Recognition with Unsupervised Feature Learning

Justin ChenStanford University

justinkchen@stanford.edu

Abstract

This paper focuses on experimenting with different seg-mentation approaches and unsupervised learning algo-rithms to create an accurate sign language recognitionmodel. To more easily approach the problem and obtainreasonable results, we experimented with just up to 10 dif-ferent classes/letters in the our self-made dataset instead ofall 26 possible letters. We collected 12000 RGB images andtheir corresponding depth data using a Microsoft Kinect.Up to half of the data was fed into the autoencoder to ex-tract features while the other half was used for testing. Weachieved a classification accuracy of 98% on a randomlyselected set of test data using our trained model. In addi-tion to the work we did on static images, we also createda live demo version of the project which can be run at alittle less than 2 seconds per frame to classify signed handgestures from any person.

Future Distribution PermissionThe author of this report gives permission

for this document to be distributed to Stanford-affiliated students taking future courses.

1. Introduction

The problem we are investigating is sign lan-guage recognition through unsupervised featurelearning. Being able to recognize sign lan-guage is an interesting computer vision problemwhile simultaneously being extremely useful fordeaf people to interact with people who don’tknow how to understand American Sign Lan-guage (ASL).

Our approach was to first create a data set

showing the different hand gestures of the ASLalphabet that we wished to classify. The next stepwas to segment out only the hand region fromeach image and then use this data for unsuper-vised feature learning using an autoencoder, fol-lowed by training a softmax classifier for mak-ing a decision about which letter is being dis-played. Many different segmentation approacheswere tried until we discovered that the skin colorand depth segmentation techniques worked mostconsistently.

2. Methodology

2.1. Problem Statement

2.1.1 Background

The CVPR gesture workshop from 2011 providesa great information on modern gesture recogni-tion models as well as how to incorporate differ-ent learning algorithms. There is some past work1

related to our project that we initially looked atsuch as segmentation-robust modeling for signlanguage recognition [3] and sign language andhuman activity recognition [1], but we ended upusing mostly our own approach to sign languagerecognition. Inspiration for our learning modelwas drawn from the MNIST handwritten digitsrecognition problem2 which also used a similarunsupervised feature learning and classificationapproach [2]. We also investigated the use ofconvolutional neural networks for feature learn-ing based on the visual document analysis paper

1http://clopinet.com/isabelle/Projects/CVPR2011/2eblearn.sourceforge.net/demos/mnist/index.shtml

by the Microsoft Research group [4]. We felt thata simpler approach, however, would still yieldcompetitive accuracy results.

2.1.2 Dataset

The data used for training and testing came fromour own self-made dataset. 1200 samples of eachof the 10 signed letters (a, b, c, d, e, f, g, h, i,l) were collected using the Kinect including thecorresponding depth data. Each sample consistsof a person signing the corresponding letter whilefacing directly at the Kinect camera. This datasetconsists of 6000 images used for training and an-other 6000 images used for testing. The followingis a visualization of some of the unprocessed rawimages in the dataset:

Figure 1: Example Image in Dataset of (Top Left) letter ’F’, (TopRight) letter ’L’, (Bottom Left) letter ’I’, (Bottom Right) letter

”D”

The depth data that was collected along withthe RGB data overlayed perfectly on top of theoriginal image and was helpful during the seg-mentation section of the model. An example isshown below:

Figure 2: Depth data overlayed on top of RGB data

2.2. Technical Approach

2.2.1 Segmentation Methods

We tried implementing many different versionsof quick image segmentation, in particular, handsegmentation.

Edge SegmentationOne of the methods we tried was using a Canny

edge detector to find relevant ”objects” in the fieldof view of the camera. The edges were then di-lated, and then all remaining holes in the maskwere filled to create a solid, continuous mask.Once this was done, only the largest areas weretaken in order to remove all the background clut-ter objects. This approach makes the simplify-ing assumption that the biggest objects seen insegmentation are typically of the most interest aswell.

Figure 3: (Left) Edge detection results, (Right) Edge detectionmask applied to image shown by black outline

However, this approach did not always worksince there would be instances where the largest”object” with distinguishable edges was not thehand (or there was a busy background which clut-tered the output) and therefore segment poorly.We decided to move on to a simpler, but perhapsmore accurate method of segmentation using justcolor.

Skin Color SegmentationWe tried out the two approaches for skin seg-

mentation using only color information. The firstapproach involved modeling the skin color bya 2D Gaussian curve and then using this fittedGaussian to estimate the likelihood of a givencolor pixel being skin. First, we collected skinpatches from 40 random images from the inter-net. Each skin patch was a contiguous rectangular

skin area. Skin patches were collected from peo-ple belonging to different ethnicities so that ourmodel is able to correctly predict skin areas for awide variation of skin color. The colors were thennormalized as follows :

r = RR+G+B

, b = BR+G+B

.

The g component is ignored as it is linearly de-pendent on the other two. The mean and covari-ance matrix of the 2D Gaussian (with r, b as theaxes) is estimated as follows :

Mean m = E[x], where x = [r, b]T

Covariance C = E[(x−m)(x−m)T ].

Figure 4: (Top) Histogram of color distribution for skin patches,(Bottom) Gaussian model fit

With this Gaussian fitted skin color model, thelikelihood of skin for any pixel of a given test im-age can be obtained. If the pixel, has a chromaticpair value of (r, b), then the likelihood of skin forthis pixel is given by:

Likelihood = e[−0.5(x−m)TC−1(x−m)], wherex = [r, b]T .

Finally, we thresholded the likelihood to clas-sify it as skin or non-skin. However, this approachdid not give significantly good or consistent re-sults and failed to detect dimly illuminated partsof skin. These poor results did not meet the qual-ity standard of hand segmentation we needed toget a consistent feature extraction from the learn-ing layer of the model.

The second approach which we used is moti-vated by the paper [5], in which the authors firsttransform the image from the RGB space to theYIQ and YUQ color spaces. The authors thencompute the parameter

Θ = tan−1(V/U)

and combine it with the parameter I to definethe region to which skin pixels belong. Specifi-cally, the authors called all pixels with 30 < I <100 and 105o < Θ < 150o as skin. For ourexperiments, we tweaked these thresholds a bit,and found that the results were significantly bet-ter than our Gaussian model in the previous ap-proach. This might have been because of two rea-sons:

1. The Gaussian model was trained using datasamples of insufficient variety and hence was in-adequate to correctly detect skin pixels of darkershades

2. Fitting the model in the RGB space performspoorly as RGB doesnt capture the hue and satura-tion information of each pixel separately.

Figure 5: (Left) Skin detected using Gaussian model, (Right)Skin detected using YIQ and YUV color spaces

Having detected the skin regions quite accu-rately, we then further filtered out the hand regionand eliminated the face/other background pixelsthat might have been detected by using the corre-sponding collected depth data. We assumed thatin any given frame, the hand was the object of in-terest and therefore the closest skin-colored objectin the camera’s view. We then created a secondarydataset out of this segmentation model which con-sisted of hand gestures for ten letters of the ASLalphabet. Each image was cropped and resized tobe a square 32x32 bounded area.

Figure 6: Visualization of processed database images

Further Segmentation WorkWe explored further segmentation methods, but

eventually deemed them of less importance thanthe original skin-color hand segmentation to getan accurate ASL recognition system. One of theother approaches was attempting to properly seg-ment pointed fingers within the detected hand. Inorder to segment out the fingers, we used convexhull detections to find the possible ”fingers” af-ter the hand has already been segmented out. Thefingers will ideally be oriented along the directionfrom the convex hull point to the centroid of thehand as seen in Figure 7.

Figure 7: (Left) Skin model segmentation, (Right) Using convexhull detection to find potential ”fingers”

However, we never ended up using the detectedfingers as part of the classification approach be-cause of the slightly unpredictable detections ofconvex hulls in a hand image.

2.2.2 Feature Learning and Classification

The extracted data of hand images were fed intoan autoencoder in order to perform the actualrecognition training part of the project. This stageimplements an unsupervised learning algorithm.We feed all the data samples into the sparse au-toencoder. The input data from the segmentationblock are images of size 32x32 pixels. A sparseautoencoder is chosen initially with an input layerwith 32x32 nodes and one hidden layer of 100nodes. We used L-BFGS to optimize the costfunction. This was run for about 400 iterations toobtain estimates of the weights. Now the autoen-coder has learnt a set of features similar to edges.A visualization of the learned features can be seenbelow:

Figure 8: Visualization of sparse autoencoder features

The next step is to classify the 10 differentletters based on the features learnt by the autoen-coder training. The output of the hidden layer ofthe autoencoder is fed into a softmax classifier

to now classify the data into 10 categories. Thesoftmax classifier again learns using the L-BFGSoptimization function. This algorithm convergesafter about 40 iterations. We tested the system

accuracy by using the remaining 600 images perletter (for a total of 6000 images) as our test set.

An overall view of the system’s block diagram can be seen here:

Figure 9: Block diagram summarizing our approach for sign language recognition

3. Experimental Results and Discussions

In the previous sections, we have mentioneddetails of our implementation of the hand segmen-tation, unsupervised feature learning and classifi-cation sub-blocks. In this section, we report theperformance of our system through tables and fig-ures. Our primary evaluation metric is based onclassification accuracy. Given an unsegmentedimage of a person signing a letter, we want to

see what the accuracy of our model is in clas-sifying/predicting the signed letter in the image.Achieving a classification accuracy around 98%similar to many MNIST digit recognition scoresis desired.

As a preliminary diagnostic, we plotted a learn-ing curve showing the training error and the testerror as a function of the size of the training set.The following plot shows the learning curve weobtained:

Figure 10: Learning curve of sign language recognition system

In our milestone report, we had used 50 hiddenunits for our autoencoder. Analyzing our learningcurve, we observe that the training error and thetest error are close to each other (except for oneaberration at training set size 3000), and even thetraining error is more than 1.5%, which is some-what high. Hence suspecting that we might be inthe high bias region, we decided to increase thesize of our features by increasing the number ofhidden units of our autoencoder to 100. Our finalclassification accuracy on the test set achieved us-ing this 100 length feature vector was 98.2%.The

following table summarizes all our implementa-tion details and reports the accuracy obtained :

Table 1: Classification Accuracy Evaluation Results on Test Set

The accuracy results that we obtained are com-parable to the accuracy of digit recognition on theMNIST dataset and therefore we believe that our

approach works relatively well.As a finishing step to our project, we have suc-

cessfully created a real time implementation ofour entire system, so that hand gestures made infront of the Kinect connected to our computer di-rectly displayed the image captured by the kinect,the segmented hand gesture and the output of our

classifier, which is one of the ten letters in ourdataset. The evaluation process takes less than 2seconds per frame. The following figures showscreenshots of our real-time implementation andthe results obtained. In each screenshot, the origi-nal image is shown with the result of the segmen-tation and the predicted result to the right of it.

Figure 11: Example screenshots of real-time live demo sign language recognition system interface

4. Conclusion and Future Work

In this project, we have implemented an auto-matic sign language gesture recognition systemin real-time, using tools learnt in computer vi-

sion and machine learning. We learned abouthow sometimes basic approaches work better thancomplicated approaches. Despite trying to use asmart segmentation algorithm, the relatively ba-sic skin segmentation model turned out to extract

the best skin masks. We also realized the time-constraints and difficulties of creating a datasetfrom scratch. Looking back, it would have beennice to have had a dataset already to work off of.Some letters were harder to classify in our livedemo such as ”a” vs ”i” since they only differ bya very small edge (the ”i” has the pinky point-ing up). Although our classification system worksquite well as has been demonstrated through ta-bles and images, theres still a lot of scope for pos-sible future work.

Possible extensions to this project would be ex-tending the gesture recognition system to all al-phabets of the ASL and other non-alphabet ges-tures as well. Having used MATLAB as the plat-form for implementation, we feel that we can alsoimprove upon the speed of our real-time systemby coding in C. The framework of this project canalso be extended to several other applications likecontrolling robot navigation using hand gesturesand the like.

References[1] D. Metaxas. Sign language and human activity recognition,

June 2011. CVPR Workshop on Gesture Recognition.[2] M. Ranzato. Efcient learning of sparse representations with an

energy-based model, 2006. Courant Institute of MathematicalSciences.

[3] S. Sarkar. Segmentation-robust representations, matching,and modeling for sign language recognition, June 2011.CVPR Workshop on Gesture Recognition, Co-authors: Bar-bara Loeding, Ruiduo Yang, Sunita Nayak, Ayush Parashar.

[4] P. Y. Simard. Best practices for convolutional neural networksapplied to visual document analysis, August 2003. SeventhInternational Conference on Document Analysis and Recog-nition.

[5] X. Teng. A hand gesture recognition system based on locallinear embedding, April 2005. Journal of Visual Languagesand Computing.

5. Appendix

This project is done in combination with theCS229 Machine Learning final project. TheCS231A Computer Vision primary component isthe hand and finger segmentation using 3D cam-era.