Virtual Hand A HCI Testbed for Computer Vision Researches on Human Hand by Beifang Yi August, 2003.

Virtual Hand

A HCI Testbed for Computer Vision Researches on

Human Hand

byBeifang YiAugust, 2003

Virtual Hand

A thesis submitted in partial fulfillment of the requirments

for the degree of Master of Science

with a major in Computer Engineering

Virtual Hand

● Objective:– To provide a virtual environment for experiments on

human hand in computer vision domain:● A dynamic hand model● A graphical user interface

Virtual Hand

● Presentation outline:– Background– Hand model– Interface design– Demonstration

Virtual Hand: Background

● Human-Computer Interaction (HCI)– A discipline concerned with the design, evaluation

and implementation of interactive computing systems for human use and with the surrounding environment.

● People● Computing system● Usability● Interaction


● Human-Computer Interaction (HCI)– On human side: human cognitive features—attention,

motivation...– On machine side: input/output devices.


● Human-Computer Interaction (HCI)– “Almost any natural communication among humans

involves multiple, concurrent modes of communications.”

– Therefore, “people prefer to interact multimodally with computers.”


● Human-Computer Interaction (HCI)– Traditional computer has made full use of its “touch”

faculty: keyboard, mouse, touchpad...– Now, HCI is trying to employ other means that

human boasts, especially, that of sight (vision) sense.


● Vision-based HCI:– By vision, we refer to the use of camera,– And a set of visual or graphical techniques for

presenting and processing information.– This visual sensing endows a vigorous input mode to

HCI.– Examles: face/gaze/fatigue/on-road-car detection...


● Hand-related Vision-based HCI:– Hand gestures, a kind of non-verbal interactions

among humans,– But have a linguistic role.


● The dexterity of human hand:– Humans have five senses, but animals enjoy far more

acute, perfect sensory faculties with exception of the sense of touch.

– “We ought to define the hand as belonging exclusively to man,” Sir Charles Bell acknowledged in 1885, because its sensibility and ingenuity “converts the being who is the weakest in natural defense, to the ruler over animate and inanimate nature.”


● The dexterity of human hand:– Our ancestors long ago

did understand this.● Left: a red hand

silhouette painted on a stone slab, about 10,000 years ago.


● The dexterity of human hand:– Do you know Chinese

Gongfu (Taiji)?● With exertion on your

hand a little force (only of several Newtons), you can fend off offender's strike of thousands of Newtons, which will bounce back upon the offender.


● The dexterity of human hand:– At present, we have

been intensively taking advantage of our hand.


● The dexterity of human hand:– From now on, we

will be depending on our hand gestures to give commands to computing systems.


● Functions of the hand:– “The hand at rest is beautiful in its tranquility, but it is

infinitely more appealing in the flow of action.”– The functions of the hand can be described through:

● Opposition● Prehensile movement● Non-prehensile movement


● Function of the hand—opposition:– The pulp surface

of the thumb is placed squarely in contact with or opposite to the terminal pads of any rmaining digits.


● Function of the hand—prehensile:– Gripping or

pinching an object between the digits and palm.


● Function of the hand—non-prehensile:– Pushing, lifting,

tapping, typewriting...


● Hand gestures:– “If language was given to men to conceal their

thoughts, then gesture's purpose was to disclose them.”

– Classification of hand gestures varies from their functionality, linguistic implication, application, and etc.


● Anatomy of the hand:– Main elements: bones,

muscles, tendons.– Tendon, the rough

cord of dense white fibrous tissue, unites muscles with bones and transmits the force which the muscles exert.


● Anatomy of the hand:– The transmission of

force produces hand movement at a joint between bones.

– We can see this movement as a result of the hand bones' positions and orientations.


● Hand as a machine:– There are reasons:– Motor: muscle, two-

stroke engine: pull-push.

– Transmission: tendon.

– Application: skin and pulp tissue.


● Hand joints:– Fingers:

● DIP● PIP● MCP

– Thumb:● IP● MCP● CMC


● Hand joints:– For simplicity,

use left diagram.– Hand is

considered as a manipulator with 27 degrees of freedom (DOFs).


● Hand joint movement:– Flexion and

extension,– Abduction and

adduction,– Rotation.


● Biomechanics of the hand:– At a joint, two

coordinate systems:

● Proximal● Distal

– Their axes' representations.


● Biomechanics of the hand:– Thus we have:


● Biomechanics of the hand:– The finger joint

motion (flexion-extension) ranges.


● Biomechanics of the hand:– The thumb joint

motion (flexion-extension and abduction-adduction) ranges.


● Hand constraints:– Usually, we study natural hand movement.– Hand can not make any arbitrary gestures.


● Some of the hand constraints:– The four fingers are planar manipulators with the

exception of the MCP joints.– There exists a relationship between DIP and PIP

flexions.– The MCP joint in middle finger displays little

abduction and adduction.


● Some of the hand constraints:– There exist “finger planes”: for each finger, the five

points(tip, DIP, PIP, MCP, wrist joint) are coplanar.– There exist a “thumb plane”: four thumb points are

coplanar.– There exists a “palm plane”: all the finger MCP joints

are in a plane, which is perpendicular to “finger plane”.


● Hand modeling:– Physical models: hand deformation under internal and

external forces applied to hand.– Statistical models: deformation “learning” through a

set of training examples.– Geometrical models: cubes, cylinders, 3D polygons,

spline-based surfaces, and etc.


● Hand modeling (example):– Multi-layered (physical) modeling: complicated.


● Hand modeling (examples in computer vision):– Geometrical models: simple.


● Hand modeling (examples in computer vision):– Geometrical models: a little sophisticate.


● Hand modeling (examples in computer vision):– Geometrical models.

Virtual Hand: Modeling

● Hand modeling:– Static hand model– Dynamic hand model– Implemented with Open (Coin) Inventor


● Static hand model:– Raw hand data acquisition:

● Simple: cyliners, transformed cubes and spheres.● Sophisticate: a set of 3D points from 3D scanner..., the

size varies.● Can be found on Internet or purchased from company.


● Static hand model:– Once raw data obtained, it's easy to reconstruct a

static hand model through the points' connections:● Normals are calculated for all points: vector addition,

multiplication(cross), normalization.● Displays rendered in different styles.


● Static hand model:– About 3,000

polygons,– Forearm

included, comfortable to look at.


● Static hand model (displayed in different styles):


● Dynamic hand modeling:– Partition the static hand model into parts according to

hand joints,– Exert motions at joints,– Integrate the hand parts as a whole hand.


● Dynamic hand modeling:– Partitioning hand (Inventor's symbols used):


● Dynamic hand modeling:– Partitioning hand to 17 hand parts with points

numbered: find and record the edge points between adjacent hand parts. Following is the thumb base.


● Dynamic hand modeling:– Hand part movement: find (calculate) rotation axes

and points for different movements.


● Dynamic hand modeling:– Now we exert motions at hand joints and end up with

“holes”.


● Dynamic hand modeling:– Hole of a hand part: produced by the trajectories of

the edge points of the hand part after it rotates to a new place.

– How to fill (patch) up the hole? To sample the trajectories to obtain some surface points of the hole.


● Dynamic hand modeling:– Now a hand part at motion is made of two parts: itself

and the patch that is used to fill up the hole.


● Dynamic hand modeling:– To produce the patching points: two new Inventor

classes are written extended from Inventor's “SoEngine”:

● One for thumb base in which some kind of deformation is considered, and

● The other for rest of hand parts.


● Dynamic hand modeling:– Integration of the hand parts and their patches for

holes:

Virtual Hand: Interface Design

● Interface design issues:– HCI's usability: easiness, efficiency and

satisfactoriness to:● Produce any hand gestures,● View from different viewing points,● Control hand joints' movements,● Record rendered images and hand configurations,● Consider hand calibration and inverse kinematics (future),● And ...


● Interface implementation:– With Qt for interface design,– Coin's “SoQt” as connection between Inventor and

Qt,– Using Kdevelop as developing environment,– Cross platform.


● Testbed's architecture:


● Performances: an overview.


● Performances: viewing (camera and display window) control panel.


● Performances: viewed from cameras with different foci.


● Performances: virtual hand in various backgrounds and rendering styles.


● Performances: thumb control panel.


● Performances: thumb in motion.


● Performances: finger control panel.


● Performances: index in motion.


● Performances: fingers in motion.


● Performances: wrist control panel.


● Performances: wrist in motion.


● Performances: hand calibration.


● Performances: inverse kinematics testing.


● Performances: constraint application(colored hand).


● Performances: animation in line styles.


● Performances: recording control panel.


● Performances: rendered with basic light model (only diffuse object color).


● Performances: animation process recording with basic light model (only with diffuse object color).


● Performances: animation process recording (rendered with Phong light model).


● Performances: animation process recording.– Not only its image sequence recorded,– But also:

● The occupied space (location and size) of virtual hand,● Camera parameters,● OpenGL rendering matrices.

Virtual Hand: Conclusions

● The virtual hand looks more realistic than the hand models used in all the computer vision projects (to my best knowledge).

● It is simpler than those deformable models and runs in real time.

● The interface design is relatively new.● Cross platform.

References

The pictures and quotations appeared in the “Background” of this presentation have been

given full citation in my thesis which is availableat

http://www.cs.unr.edu/~b_yi/vHand/thesis.pdf

Acknowledgement

Dr. HarrisDr. Nicolescu

Dr. DankoDr. Bebis

NASA/UNR project at (http://www.cs.unr.edu/~aerol/projecthome/)

Virtual Hand

Questions

?Thanks!!!

Virtual Hand A HCI Testbed for Computer Vision Researches on Human Hand by Beifang Yi August, 2003.

Documents

Transcript of Virtual Hand A HCI Testbed for Computer Vision Researches on Human Hand by Beifang Yi August, 2003.