8/10/2019 DataGlove+CHI+1987

1/4

CHI GI 1987

A H A N D G E S T U R E I N T E R FA C E D E V I C E

Thomas G . Zimmerman, Jaron Lanier,Chuck Blanchard, S teve Bryson and Young Ha rvill

VP L Research, Inc.656 Bair Island Road, Suite 304

Redw ood City, CA 94063

A B S T R A C T

This paper reports on the development of a hand tomachine in terface device that provides real - t imegesture, position and orientation information. The keyelement is a g love and the device as a wholeincorporates a collection of technologies. Analog flexsensors on the glove measure finger bending. Handposi t ion and or ienta t ion are measured e i ther byultrasonics, providing five degrees of freedom, ormagnetic flux sensors, which provide six degrees otfreedom. Piezoceramic benders provide the wearer ofthe glove with tactile feedbac k. These sensors aremoun ted on the light-weight glove and connected to thedriving hardware via a small cable.

App l i ca t ions o f the g love and i t s componen ttechnologies include its use in conjunction with a hostcomputer which dr ives a real - t ime 3-dimensionalmode l of the hand a l lowing the glove wearer tomanipulate c omputer-gen erated objects as if they werereal, interpretauon o f finger-spelling, evalua tion ofhand impairment in addition to providing an interfaceto a visual programming language.

Resume

Cet article presente le developpement d'un systememain-machine qui foum it en temps reel les positions,orientations et gestes de la main de l 'utdisateur.L element de base du systeme est un gant, l 'ensembledu systeme util isant differentes technologies: Descapteurs a nalogiq ues pla ces sur le ~,ant mes urent lesflexions des doigts. La position et l 'orientation d e lamain sont mesures, soit a 1 ' aide d'un systemeultrasonique, qui fo umit cinq des degres de l iberte dusysteme, soit a l 'aide d'un systeme magnetique, quifourni t es s ix degres de l iber te . Des e lements

ieZoceramiques places sur le gant sont controles poururnir a l 'utilisateur differentes sensations tactlles.

Ces differents capteurs sont montes sont un gant defaiblepoids, connecte au systeme de controle par uncable de faible dimension.

Les app l i ca t ions du sys teme comprennen t sonutilisatlon relie a un ordin ateur conte nant un mod eletemps reel entr0is dimensions de la m ain pe rmettanta l 'util isateur de manipuler des objets virtuelsgeneres par l 'ordinateur comme s' ls existaient reelee-

ment, l interpreta,fibn de signes alphabe tiques pou rmal-entendants, 1 eyaluatioia clinique d e troub les f onc :tionnels de la main, ainsi qu son util is ation com meinterface a un langage de programrnationvisuelie.

Keywords: Human Interface, User Interface, MotorInterface, Tactile Interface, Gesture R ecognition.

1 . I N T R O D U C T I O N

The hand gesture input devices presented here, theZ - G l o v eTM and the DateGloveTM, are l ightweightcotton gloves containing flex sensors which measurefinger bending, positioning and orientation systems,andtactile feedback vibrators.

Permission to copy without fee all or part of this mater ial is grantedprovided that the copies are not m ade or dis tr ibuted for directcommercial advantage, the AC M copyright not ice and the t i t le of thepublicat ion and i ts date appear, and not ice is given that copying is bypermission of the Associat ion for Comp uting M achinery. To copyotherwise, or to republish, requires a fee and/or specific permission.

1987 ACM-0-89791-213-6/87/0004/0189 $00.75Figure 1.

DataGlove w ith outer glove removed to show sensors

89

8/10/2019 DataGlove+CHI+1987

2/4

CHI + G I 1 9 8 7Different or ientat ion and posi t ioning systems of theZ-Glove and the DataGlove dis t inguish the twomode ls . The Z-Glov e employ s an u l t r a son ic

osi t ioning and or ientat ion system control led by aommodore 64T M computer and is less costly and

mo re limited in applicauon. The DateGlove, using amagnetic positioning and orientation system controlledby an Apple MacintoshM compu ter is mo re expensiveand o f wider application (see Fig. 1).

2. E L E M E N T S O F T H E S Y S T E M

2 .1 F L E X S E N S O R S

On both glov es, the flex sensor is a patented [Zim85]opt ical goniometer that can be manufactured in avariety of dimensions suited to the application. Flexsensors measuring two inches long by one quarter inchin diam eter are glued to a tight-fitting, stretchable inne rglove. Multiplexing electronics are mou nted on top ofthe inner glove andpr otected by a loose-f i tt ing outerglove.

Five to fifteen flex sensors are mounted on a glove,depend ing on the appl ication. The f inger jointsmeasured are the metacarpophalangeal (MP or inner)joints , theproximal interphalangeal (PIP or middle)joints an d their abductions, and the palm (see Fig. 2).

~ O U d T ~ O ~

~- 2 a-LoVe ~ ~ I

k F 2 2 2 _ _ lFigure 2

Flex and ultrasomc transducer placement

On the DataGlove, analog signals fro m the flex sensorsare mult iplexed to data-acquis i tion hardware whichinterfaces to an Apple Macintosh computer. On theZ-Glove, analog flex sigrrals are fed directly to theintemal converters of the Com mod ore 64 computer.

2 .2 P O S I T I O N I N G A N D O R I E N TAT I O N

2 .2 .1 U LT R A S O N I C P O S I T I O N I N G N DO R I E N TAT I O N

The positioning and orientation system on the Z-Gloveconsis ts of two ul t rasonic t ransducers a t tached toopposite sides o f the metacarpal (between the wrist and

the . f ingers) , a hardw are t imer, three ul t rasonicrecewers mounted around the per imeter of a moni torand t r iangulat ion sof tware runnmg on a Com modo re64 computer. Processing overhead is reduced byusin g triangu lation approx~rnations. Visu al feedb ack 1provided by an icon which t racks hand movement ,minimizing the impact o f tracking errors.

Rol l and yaw of the hand is determined from theposi t ion of the t ransmit ter pair. The ul t rasonic"transmitters need a direct line-of-sight for their acoustisignals to reach the receivers. If one transmitter isblocked (by the fingers or part o f the hand), orientatioinform ation is lost. If both transmitters are blockedthe tracking icon freezes.

Position ji t ter due to detection error is reduced, withmin imu m imp act on response t ime, by. means of asof tware f i l ter that decreases the f i l ter ing withincreasing hand velocity.

2 . 2 . 2 M A G N E T I C P O S I T I O N A N DO R I E N TAT I O N

On the Da taGlove , a 3SPACET M ( P o l h e m u sNavigat ion Sciences Divis ion, McDonnel l DouglasElec t ron ics Company, Esses Junc t ion , Vermont )position and orientation system is used. The 3SP ACEuses low frequen cy magnet ic f ie lds to measur e s ixdegrees of freedom. The smal l 3SPACE sensor ismou nted on the dorsal s ide of the hand between theglove's two layers. The 3SPA CE is conne cted to onof the serial ports on the App le Macintosh co mputerThe 3S PAC Erequ ires no fil tering of data.

2 . 3 TA C T I L E F E E D B A C K

Tactile feedback is used to add realism to interactionbetwe en comp uter-gen erated (virtual) objects and thvirtual hand. Tactile cues are being exp erimented withto s imulate object contact , hardness and surfacetexture.

Piezoceramic benders dr iven with a 20-40 Hz s inewave, (b elow peak sensitivity of 250 Hz [Sherrick82to pre ven t audible sound generat ion) are moun tedunderneath each f inger ; The sensat ion produced i"tingling" or "num bness ' .

Benders ar e used because of their small size, low cosand low operating voltage. Freque ncy mod ulation iused to var y the intens ,ty of the tactile sensation and tminim ize the f inger 'num bing " sensat ion. Tact i lestimulation is increased by driving ben ders closer topeak tactile sensitivity frequen cy. Object contact iscued when vir tual f inger t ips touch the surface ovirtu al objects,. Co nta ct is signale d by ,,oscillatingfinger benders, which prod uce a "buzzing sensationat the fingertips.

3 . G E S T U R E R E C O G N I T I O N A N DC A L I B R AT I O N

Representing and recogn izing hum an hand gestures ia deep problem analogous to the recognition of huma n

190

8/10/2019 DataGlove+CHI+1987

3/4

C H I + G I 1 9 8 7

speech or handwri t ing. W e present here some of thebasic method s o f gesture recognition.

It is possible to iden tify two types of gestures: objectmanipulations (eg: pick up, rotate, throw, squeeze),and com man ds (eg: draw a line, produce a sound, set acolor). Gestur es can also be classified as static (eg: a"victory" or a "peace" sign), or dy nam ic (eg'. wav mggood-bye).

Z-Gloves and DataGloves are manuf actured in severa]sizes taking into account the wide variety of humanhands. Individu al gesture styles further comp licategesture recogni t ion. Two techniques of cal ibrat ionhave been deve loped to compensa te fo r thesevariations.

3 . 1 M A N U A L C A L I B R AT I O N

Manu al cal ibration is a method in which a subjectwearing the Z-Glove performs tasks . One manualcahbrat~on me thod requ ires the Z-glove wear er to graba ser ies of objects placing f inge r joints a t know nangles. The joint bending angles recorded in thismanu al calibration are used in a clinical study of handrehabilitation.

A finger spelling interpretation application uses m anualcal ibrat ion to form gesture templates . Gesturerecogni t ion is performed by different ia t ing f ingervalues with values from such gesture templates. Theabsolute value o f these differences is sum med for eachgseSture template. The gesture with the m inim um su m

chosen ff that sum is less than a confidencethreshold. I f the gesture does not match any of thetemplates closely enough then the gesture is ignored.

3 . 2 A U T O M AT I C C A L I B R AT I O N

Gestur e recognition using autom atic calibration scalesf l ex sensor va lues by record ing mimimum andmaxim um jo in t ex tens ion va lues . The maximumvalues are decremented over t ime to prevent an extremejoint extension from skewing the scaling of recordedvalues. Hysteresis thresholds unique to each finger artappl ied to reduce the posi t ion of the f ingers toquan tized states. Th ese finge r states are then comparec[to gesture templates containing permissable f ingerstates an d the first succes sful gesture template match ischosen , o r " recogn ized" . Th resh o ld va l ues andges tu re t empla tes a re use r- independen t and a reempirically derived.

, ,Unintentional passin g (transition al) g estures a redebounced" by requiring a valid gesture to be h eld for

a period o f t ime.

Dynamic gesture recogni t ion techniques are beinginvest igated. As in speech recogni t ion, dynam icgesture recognition is able to take advantage of con textm order to l imit the number of gestures to bedistinguished at a giv en time.

4. A V I S U A L P R O G R A M M I N GL A N G U A G E I N T E R FA C E

A Z -Glove based u ser interface is explored in a visualp r o g r a m m i n g e n v i r o n m e n t k n o w n a s G r a s pT M(originally called Mandala) un der developm ent at VPLResearch, Inc. [Lanier84]. The Grasp system runs onmicrocom puters and uses the ul t rasonic posi t ioningand orientation system.

A m oving hand-shaped icon t racks the user 's hand.W hen a val id gesture is detected, the hand image isshown perform ing that gesture . This produces adiscrete animation of a hand, rather than a continuousrepresentation of the hand.

In mos t mouse (o r o the r po in t ing dev ice ) basedinterfaces, there are two phases to each user action:selection of an object, and selection of the operation tobe perfor med on the selected object. In Grasp, objects e l e c t i o n a n d o p e r a t i o n a r e a c c o m p l i s h e dsimultaneously by gesturing ov er an object.

In order to make gestures easier to learn andrem emb er, gestures n G,r, sp are an alogous toreal-world gestures. A gra b' ( all f'mgers closed like afist) picks up ~n object. Onc e picked up, the object carbe carried around the screen and 'dropped" (all fingersopened) at a new location. If the user mere ly opens afew fingers, a copy of the object is put down. If theuser picks up an o,bject wi~, on ly the thumb and indexfinger, a value is ' plucked from the object. Openingfingers quickly over an objects opens or expands theobject.

5 . A P P L I C AT I O N S

Am ong the present applications of the DataGlove andthe Z-G love are a gesture recogn ition device, a clinicaltool for evaluating hand functlon, a three-dimensionalhand model control ler, an interface to a visualprogramm ing language, a music and sound synthesiscontrol ler, a f inger spel l ing interpreter, and acompu te r-genera ted ob jec t manipura to r. Fu tu reprojections for DataG love and Z -Glove applications liem the fields of robotics, hum an factors ander gono micsresearch.5 .1 T H R E E - D I M E N S I O N A L H A N D M O D E L

A flexible, three-dimensional, articulating hand modelis constructed using local coordinates for each fing erjoint, represented by a l inked list of records, with eachfinger having a full range of angular motion. The handmod el (virtual hand) is generated in real t ime using theDateGlove and the Apple M acintosh computer (see

photo).5 .2 A C L I N I C A L H A N D I M PA I R M E N T

M E A S U R I N G T O O L

Hand impairment measurements are c lass i f ied asanatomical and funct ional. A goniom eter obtainsanatomical measurements by measuring the range of

9

8/10/2019 DataGlove+CHI+1987

4/4

CHI GI 1987

motion o f a given joint. Functional tests measure thetime it takes to perform comm on unilateral tasks l ikestacking check ers, turning ove r cards, or putting smallobjects m a can [Jebsen69].

The p.rocess of measuring the range of motion of apatient s hand b y a skilled therapist with a mechanicalgoniometer can take one tO two hours and is onlyrepeatable to within five degrees if the same physicaltherapist with a mechanical goniometer performs themeasurements [Rosen86] . Th e DataGlove has thecapability of measuring a patient's range of motion in afraction of the t ime, under the supervision of a lessskilled assistant, with mor e repea table results. Testsare now b eing co nducted to exam ine this application.

Functional tests l ike the Jebsen Hand Function Testtake a considerable amount of t ime for the patient andthe physical therapist. Dyna mic data recorded on videotape is a lso t ime consu ming to analyze. TheDataGlove, with a positioning and orientation system,is being invest igated as a means of logging andanalyzing fun ctional tests.

5.3 T H R E E - D I M E N S I O N A L O B J E C TM A N I P U L AT I O N

Manipulating three-dimensional virtual objects withtwo-dimensional controllers such as digitizing tablets,touch pads and mice are awk ward since these objectsare capable of six-dimensional move ment. Both theZ-Glove and DataGlove (wi th a posi t ioning andorientation system) allow users to interact with virtualobjects much as they do with real objects. Virtualobjects can be pickedup, grabbed, twisted, squeezed,thrown, and set down.

A virtual object can be m oved (translated and rotated)with either the Z-Glove or the DataGlove while i t isbeing operated on (eg: squeeze d and twisted). Andsince simultaneous actions are usually faster thansequential ones [Buxton86], the entire function can beperform ed in a more natural, more efficient manner.

The Z-Glove and the DataGlove address the questionof "scoping", the selection of a subset from a set.This is significant when several objects appear near oion each other. Consider a computer-generated imageof cherr ies in a glass bowl viewed from the s ide .W hen using a mou se to point a t a cherry, it i sambiguous whether o ne is pointing to the cherry or thebow l. In contraast, one cou ld use the DataG love or theZ-Glo ve to close one's fingers around and pick up thedesi red i tem, wi thout ambigui ty. A handful ofcherries, for that matter, could be picked up, thenumbe r of cherries in the handful being a function ofthe size of the grasp.

A virtual environmentproject at NASA A mes Researc hC e n t e r [ F i s h e r 8 6 ] u s e s a h e a d - m o u n t e d ,t h r e e - d i m e n s i o n a l d i s p l a y [ S u t he r l a n d 6 8 ] , aDa taGlove , and a 3SPACE TRACKER in cu r ren tresearch directed toward eve ntual control of a remoterobot hand. The hand performs operat ions in anenvironment hostile to human life (in space or underconditions which , due to radiation o r other factor s, is

dangerous), while the operator remains protected andcon~fortable (inside a space capsu le or safe enclosure) .

6 . C O N C L U S I O N

Two systems, the DataGlove and the Z-Glove, havebeen presente d both of which a l low the di rectmanipulation of computer-generated objects.

The best type of in terface device performs i ts taskunobtrusively. A joyst ick is a controller that a useracts upon. The DataG love and the Z-Glove, on theother h~_d, are articles of cl?thing w hich instrumentthe user s actions. The us er s hand is the controller,as is natural fo r it to be.

Just as speech is our natural mea ns of communication,the human hand is our natural means of manipulatingthe physical world . As computer sys tems begin tosimulate the physical world, the technolog ies presentedin this paper suggest a broad spectrum of possibili tiesto a wide variety of users. It is increasingly importanlthat we shape the simulated world of the compu ter inways which reflect our human universe, rather thanallow ourselves to be shaped by our m achines.A C K N O W L E D G E M E N T S

The authors would like to thank VPL Research, Inc.for its support, Sam Wanlman for early brainstorms onthe Z-Glove, Jose ph M. Ro sen, M.D. o f the StanfordUniversity Medical Center, and S cott Fisher of NAS AAm es Res earch C enter for their research contributions,and Marvin M. Zanderer for reviewing drafts of thispaper.

R E F E R E N C E S[Buxton86] Buxton, William and Myers, Brad A. p. Study

in Two-Handed Input, ACM SIG CHI 86 ConferenceProcee dings, pp.321-326 , April 198 6

[Fisher86] Fisher, Scott S. Teleoresen~ emaster alovecontroller for dexterou s robotic end-effectors. SPIECambridge Symposium on Optical and OptoelectronicEngineering, 26 October 19 86, Cambridge, Mass.

[Jebsen69 ] Je bsen, R.H. et al, Archives of PhvsicalMed icine and Reha bilitati0rl 50: pp. 311 -316, June1969

[Rosen8 6] Rosen , Dr. Joseph M., Persona l conversa tion,Septem ber 1986

[Lanier84] Lanier, Jaron, Scientific Ame rican,c o v e rSeptem ber 1984

[Sherrick82] Sherrick, Carl E., and Craig, James C.,Tactual Pe rceotion: A Sou r~ebo ok. (Schiff, W. andFoulke, E., Editors), Ca mbridge: Camb ridgeU n i v e r s i t yPress, 1982

[Sutherland68 ] Sutherland, Ivan E. A Hea d-Mou ntedThree-D imensiona l Disolav. Fall JointComp uter Conference 1968,pp. 757-764

[Zim85] Ootical Flex ~en sor, Septem ber 17, 1985,United States Pa tent Number 4,542,291

92