Hand Gesture Recognition

By Jonathan Pritchard

Outline • Motivation

• Methods

o Kinematic Models

o Feature Extraction

• Implemented Algorithm

• Results

Motivation • Virtual Reality – Manipulation of virtual objects with

one’s hands.

• Robotics/Telepresence – Precise control of

machinery from remote locations.

• Sign Language – Help the disabled interact with

computers. ASL can be used as test bed for

different algorithms.

Murthy, G. R. S., & Jadon, R. S. (2009). A review of vision based hand gestures recognition. International Journal of Information Technology and Knowledge Management, 2(2), 405-410.

Kinematic Models • Simplifying assumptions about hand motion used to

limit the degrees of freedom in the model

• Many model based approaches use a form of

causal tracking to ease computation.

o Filtering used to estimate state (pose, gesture covariance

matrix) based on previous state(s)

• Wire Frame and Silhouette models

J. M. Rehg and T. Kanade. “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J.-O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision, volume II of Lecture Notes in Computer Science 801, pages 35–46. Springer-Verlag, May 1994.

Kinematic Models: Wire Frame

J. M. Rehg and T. Kanade. “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J.-O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision, volume II of Lecture Notes in Computer Science 801, pages 35–46. Springer-Verlag, May 1994.

Stereo Vision - Hand Features Identified

Pose Estimation

3D Model

Filtering

Kinematic Models: Silhouette

Stenger, B., Mendonca, P. & Cipolla, R. “Model-Based 3D Tracking of an Articulated Hand”. In IEEE Conference on Computer Vision and Pattern Recognition, (2001) 310–315.

Silhouette matched to gesture outline with error minimizing

Kalman Filtering

Silhouettes From 3D Model

3D Model (Truncated Quadratics)

Feature Extraction • “Getting your man without finding his body parts”

• Low level image features used to extract

information without estimating pose

• Not nearly as robust as model based approaches,

but far simpler and faster to compute.

R. Polana and R. Nelson, “Low level recognition of human motion”, in Proc. of IEEE Workshop on Motion of Non-Rigid and Articulated Objects, Austin, 1994, pp. 77–82.

Feature Extraction: Number of Fingers

New, J. R., Hasanbelliu, E. and Aguilar, M. “Facilitating User Interaction with Complex Systems via Hand Gesture Recognition.” In Proc. of Southeastern ACM Conf., Savannah, (2003).

Threshold applied to saturation space, only largest connected contour kept

Image separated into HSL color spaces

Wrist removed, centroid calculated Circle centered at centroid used to calculate number of fingers

Feature Extraction: Fingertips

J. Raheja, K. Das & A. Chaudhary “Fingertip Detection: A Fast Method with Natural Hand“. International Journal of Embedded Systems and Computer Engineering , Vol. 3, No. 2, July-December 2011, pp 85-88

Orientation found by comparing oriented

histograms Fingertips detected through

algorithm looking at top edge of hand, and it’s

derivative

HSV color space used to obtain binary image

Implemented Algorithm • Fingertip Detection using MATLAB image processing

toolbox.

• Combination of previous feature extraction

algorithms

o HSV color space used to threshold binary image

o Detect orientation of hand, find outline of top

o Use outline values, derivative filter, and knowledge of hand

orientation to locate fingertips.

Preliminary Results

Binary Image

Result

Top Outline

Fingertips

Preliminary Results

Binary Image

Top Outline

Fingertips Result

Preliminary Results

Binary Image

Top Outline

Fingertips Result

Preliminary Results

Binary Image

Top Outline

Fingertips Result

Continued Work • Orientation invariance through wrist detection

• Calculate centroid of binary image

o Reject detected fingertips that are too close to centroid

• Subtract wrist for more accurate centroid

calculation.

Questions?

M. Randall “Questions”, XKCD, no. 1256 Available: http://imgs.xkcd.com/comics/questions.png