Usability Study of the Finger Counter, a Human-Computer Interface

Christen Ng and Jessie Burger

CRA DMP Project at Boston University

Abstract

The Finger Counter is a computer-vision system that identifies the number of fingers held up in front of an inexpensive webcam in real time. Usability studies were conducted to determine the limitations of the system. Tests performed included: a voice-interactive test, a drawing test, and a questionnaire that gauged the ease of use of the system, as well as a limitations test that established conservative functional ranges for use of the Finger Counter.

Introduction

Finger Counter is a computer-vision system that counts the numbers of fingers held up in front of a video camera in real time. The system is designed as a simple and universal human-computer interface: potential applications include educational tools for young children and supplemental input devices, particularly for persons with disabilities. The interface is language independent and requires minimal education and computer literacy. Finger Counter uses background differencing and edge detection to locate the outline of the hand. The system then processes the polar-coordinate representation of the pixels on the outline to identify and count fingers: fingers are recognized as protrusions that meet particular threshold requirements. The system also logs the frequency of different inputs over a given time interval. The Finger Counter interface was implemented under Linux using Video4Linux. The system was tested extensively under various lighting and background conditions to find the most favorable environment for the Finger Counter to function. Such varying conditions were also conducive to establishing limitation criteria for the system. The following is a compilation of the various tests conducted, their description, and the results obtained:

I. Tests Performed

EXPERIMENT 1 - Voice Interactive Test

Demonstration and Explanation

Test subjects were given a brief introduction to the Finger Counter: how it works, its intended purpose, the types of tests they would complete, and a demonstration on how to use it. They were then given the opportunity to familiarize themselves with the program by playing around with it before the formal testing began. During this trial period, advice was given to the participants as to how to most effectively use the Finger Counter (e.g. hands parallel to camera, maximizing the

width betOnce theon the fominutes. given onc

What foluser. Theright winshows ho

The test sbeyond inhand posresponse compliedignored brandom tasked to failed to

Test Adm

tween their fe subjects felllowing testWhen test sce.

lows is a scre window dirndow shows ow many fin

system was nitiating the

se or type thetime in seco

d with by theby the systemtime intervalremove theicomply with

ministration

fingers, orienlt sufficientlys were recorubjects were

reenshot of trectly belowraw video. Tgers the syst

completely aprogram. Fo

e key on the onds; the loge deadline (thm. After a rel between 4 ar hands fromh this reques

n

ntation and ry comfortablrded. The deme part of a gr

the test in acw it shows theThe bottom rtem recogniz

automated anollowing a rkeyboard; th

g files also dhe failures).

ecognition wand 10 secon

m the field ofst.

rotation of hle interactingmonstrationroup, the dem

tion. The tope frame rate right windowzes and the l

nd did not reequest, a usehe system loetermined thMisidentifie

was made or tnds before mf view betwe

hand placemeg with the pr

n and explanamonstration

p left windowand other st

w is the "uselocation of th

equire actioner was givenogged the reqhe number oed poses or ithe timer ran

making anotheen tests; oft

ent in regardrogram, theiation took beand explana

w is for mestatus informaer feedback wheir fingertip

n from the adn up to 5 secoquest made af requests thincorrect keyn out, the sysher request. Utentimes, ho

ds to the camir performanetween 2 to ation was on

ssages to theation. The towindow," whps.

dministratoronds to makand the user'hat were not y presses westem waited Users were wever, they

mera). nces 5

nly

e op hich

r ke the 's

ere a

Each user was permitted some time to play with the drawing program before the test administration. There was no time limit; users took from 30 seconds through approximately 3 minutes to try out the interface before going on to the test. In the "voice-interactive test," the computer plays a recorded message asking the user to hold up a certain number of fingers or else type a key from "1" through "5" on the keyboard. The audio message is supplemented by a screen message in large type. A typical message might say "please hold up one finger" or "please type 2 on the keyboard." To choose which message to play, the system generated a random number from 1 through 10. Numbers from 1 through 5 would cause the system to ask for as many fingers. Numbers from 6 through 10 would cause the system to request keys "1" through "5" on the keyboard. Random numbers were chosen "with replacement," so a single test subject could be asked to hold up some hand poses, or type some keys, a number of times and never be requested to hold up other hand poses, or type other keys. There was no set number of requests; some users got as few as 10 requests, while one got 33 requests. The test system was automated and users were given up to 5 seconds to respond.

The Test Subjects

There were a total of 19 test subjects: 6 high school students, 4 college students, 7 graduate students, and 2 professionals. The majority of college students, graduate students, and professionals were acquaintances of the administrators. The high school students were participants in a math and science summer camp. Many participants had strong technical backgrounds, as their computer experience ranged from 6 to 30 years and daily computer use ranged from 1 to 15 hours. Each test subject completed exactly one test.

Test Conditions

Multiple locations were used to conduct testing. The graduate students and professionals were tested under florescent lighting, with the camera on a tripod looking up at a ceiling about two meters above. The high school kids were tested in the graduate lounge, under florescent and incandescent lighting, with the camera mounted on a tripod facing down half a meter from the table. Subjects made hand gestures in the air above the camera when it was facing upwards and on the table surface below the camera when it was facing downwards. The program ran on a Toshiba Satellite Pro 6000 with an Intel Pentium III Mobile 1.2 GHz processor and 512 MB of RAM. The Pentium III has a bus speed of 133 MHz.

EXPERIMENT 2 - Drawing Test

Demonstration and Explanation

The same test subjects participating in Experiment 1 performed this test, with the addition of five more college students. They received the instructions described above. In addition, they were told about the "drawing test," in which they would be asked to trace a circular template on the screen, first using the Finger Counter interface and then using a computer pointing device.

Test Administration

Each Expinteractivvoice-int

The drawfingers. Thow manfinger, thcorresponfinger, thdraws a bvaries wisystem dboxes arecamera isblack. Th

For the "of a whitmatch thecolored bpressed atemplate,way aroureported the circul

The follothe circul

periment-2-tve test. Thusteractive test

wing programThe programny fingers arehe system dranding to the

he boxes maybox centeredith the distanraws three oe held up, ths processed. his keeps the

drawing testte circle. Eace circular temboxes lined ua key on the , and the test

und, the test the average lar template.

owing is a sclar template.

test subject ts, each user at before takin

m permits a um works as fo

e held up, it aws a small,position of t

y appear as ad on the midpnce between or four smalle system eraAlso at each

e screen from

t," instead ofch test subjecmplate. A teup with somekeyboard tot subject atteadministratodifference b.

creen shot of. The right w

took the drawalready had png the drawi

user to "drawollows: The tracks the fi

, randomly cthe fingertipa line or othepoint betweethe fingertip boxes corre

ases the screeh frame, the

m becoming

f being compct was asked

est subject fire point on th initiate the

empted to traor pressed anbetween each

f the drawingwindow show

wing test immplayed with ing test.

w" on the comscreen is iniingertip posicolored box (p in the field er object. If en the two fips. If the useesponding toen. New boxsystem fadeoverly clutte

pletely blackd to draw a crst moved hihe circle outltest. The scrace the circlenother key, hh pixel in the

g test. The lews the "user

mediately afthe drawing

mputer screeitially black.itions. If the (essentially aof view. Astwo fingers ingers; in thier holds up tho the fingertipxes are drawes each colorered with co

k, the initial circle with hiis or her fingline. Then, oreen cleared,e. When the halting the tee user's draw

eft window sfeedback" w

fter taking thg program an

en by movin Once the syuser is hold

a large pixels the user moare held up, is case, the shree or four p positions.

wn as each frared pixel a li

olors.

screen contais or her indegertip until thone of the tes, except for ttest subject

est. At that pwing and the

shows the uswindow.

he voice-nd taken the

ng his or her ystem determding up one l) on the screoves his or h

the system size of the bofingers, the Finally, if fiame from thittle toward

ained the outex finger to he resulting st administrathe circular made it all t

point, the sysnearest poin

ser's drawing

mines

een her

ox

ive he

tline

ators

the stem nt on

g and

After the "drawing test" with the Finger Counter, another drawing test was conducted using a computer pointing device. Available devices included a laptop touchpad, a laptop stick pointer, and a trackball pointing device. Each user was asked to choose the pointing device most unfamiliar. As with the Finger Counter drawing test, the screen displayed a circular template to which a test subject maneuvered the screen pointer (a commonly used arrow). Once the arrow was pointing at part of the template, one of the test administrators began the test, at which point a series of small colored boxes trailed the screen pointer as the user traced the circular template. When the test subject had circumnavigated the template, the administrator stopped the test. The system then computed the average distance using the same method as described above.

The Test Subjects

The test subjects for Experiment 3 included those described above in Experiment 2, as well as an additional five college students. As with Experiment 2, each test subject completed exactly one test. Three college students opted to use the AccuPoint input device as their unfamiliar pointing device, one subject did not take this test, and the remaining elected to use the trackball.

Test Conditions

The test conditions were the same for the previously listed subjects. The three additional college students were tested in their apartment under incandescent lighting, with the camera mounted on a mini-tripod facing down, approximately half a meter above the table. One of the test subjects completed the test under natural lighting conditions, aided by a portable lamp, with the camera mounted on a tripod facing upwards.

EXPERIMENT 3 - Questionnaire

The test subjects who participated in Experiments 1 and 2 were asked to complete the following questionnaire. Test subjects took from 1 to 5 minutes to do so.

Name ___________________________________________________

Telephone Number or Email (in case we have follow-up questions) __________________________

Occupation _______________________________________________

How many years of computer experience do you have? years

How many hours per day do you use a computer (on average)? hours

How easy did you find a computer mouse to use (1 = very hard 10 = super easy)? 1 2 3 4 5 6 7 8 9 10

How natural to use is a mouse (1 = completely unnatural 10 = completely intuitive)?

1 2 3 4 5 6 7 8 9 10

We asked you in the test to use a pointing device you were 1 2 3 4 5 6 7 8 9

unfamiliar with. How easy did you find the other pointing device to use (1 = very hard 10 = super easy)?

10

How natural to use is the other pointing device (1 = completely unnatural 10 = completely intuitive)?

1 2 3 4 5 6 7 8 9 10

How easy did you find the Finger Counter to use (1 = very hard 10 = super easy)?

1 2 3 4 5 6 7 8 9 10

How natural to use is the Finger Counter (1 = completely unnatural 10 = completely intuitive)?

1 2 3 4 5 6 7 8 9 10

We would appreciate any comments you might have on the Finger Counter.

In particular, what types of applications do you think would be useful with the Finger Counter?

Do you have any comments, criticisms, or suggestions regarding the Finger Counter or this testing procedure?

Thanks for your time.

EXPERIMENT 4 - Limits of Pose Recognition For this test, we placed a webcam in settings similar to the previous experiments, i.e., in the graduate computer lab, mounted on a tripod which was placed on the floor with the camera facing upwards. We ran the interface and had it capture frames as well as report poses recognized. Hands were moved in the following ways:

1. Toward the camera 2. Away from the camera 3. Rotation in the direction of each Euler angle on axes going through the center of the palm

a. "Pitch": The hand is rotated so the fingertips are closer to the camera than the palm and vice versa.

b. "Roll": The hand is rotated so that the side of the hand including the base of the little finger is closer than the side including the base of the thumb and vice versa.

c. "Yaw": The hand is rotated in the plane parallel to the image plane, so that, from the camera's perspective, the fingertips move from side to side while the palm remains more or less fixed.

A digital video camera was set up on a tripod next to the webcam and was used to capture still images of hand positions for "out of plane" rotations, i. e. pitch and roll. A hand was placed over the camera so that the hand was oriented parallel to the camera lens, perpendicular to the bottom of the image frame, and the system properly recognized the hand pose. All tests (distance, pitch, roll, and yaw) were performed on the 3 test administrators using all 5 hand positions. To determine the limits on the range of the distance of the hand position, the hand was moved

toward thhand poshand pospositionscamera uthe close To determremaininthese extthe last "measuredthe hand,hand posor 5 fingenegative.

To determeither clowas keptwas set ucould be the Fingepositionsthe horizthe pointaverage fas positiv

he camera unsition just besition just afts was also muntil recognitst "bad" han

mine the ranng parallel toremes at thegood" positid from the ve, to a line segsitions using ers. The yaw.

mine the ranoser to or far perpendicul

up so that theclearly dete

er Counter ans were then aontal axis, w where the thfingertip posve.

ntil recognitifore failure oter failure of

measured usintion failed, a

nd position, w

nge of the yaw the camera

e point just bions were theertical axis ogment drawn1 finger, an

w angle in th

nge of the pitrther away frlar to the bote camera lenrmined. Imand the digitaanalyzed usinwhich was alhumb joins tsition was us

ion failed. Aof the systemf the system,ng a tape meand two imagwere recorde

w of hand polens, until re

before recognen analyzed of the imagen from the cd to the tip oe right direc

Yaw

tch of positiorom the camttom of the i

ns was parallages of the "gal camera. Thng a softwarlong the defathe hand to tsed. Downw

An image wam, i.e. the clo i.e. the farthasure. The hges, one of ted, as well as

ositions, theecognition fanition failureusing a soft, which was enter of the of the middlection was reg

w Example

ons, the handmera lens than

image, until el to the sidegood" and "bhe images frre angle meaault hand posthe fingertipsards pitch w

as taken, usinosest possibhest "bad" phand was thethe farthest "s the distanc

e hand was rofailed. An ime and the poitware angle m the same aspalm to the e finger for hgarded as po

d was tilted n the palm, wrecognition e of the handbad" positionrom the digitasuring tool. sition, to a lis. If the fing

was regarded

ng the Fingele "good" poosition. The

en moved aw"good" handces.

otated left anmage was tak

int just aftermeasuring tos the default tip of the indhand positiositive, and th

so that the fiwhile the hanfailed. The

d and the angns were capttal camera oAngles wer

ine segment gertips were

as negative,

r Counter, oosition, and t distance at

way from thed position, on

nd right, whiken of both o. The imageool. Angles worientation odex finger fo

ons using 2, 3he left as

fingertips wend orientatiodigital camegle of pitch tured using b

of the last "gore measured

which ran fnot even, the, upwards pi

of the the these

e ne of

ile of s of were of or 3, 4,

ere on era

both ood" from

from e itch

To determcircular fthe cameup so thaImages odigital caanalyzedwhich waof the hanindex finknucklesa referenpositive.

II. Test

EXPERI

The follo

mine the ranfashion (so thera lens and pat the lens waof the "good"amera. The imd using a softas along the nd from the

nger if less th or middle oce point. Le

Results

IMENT 1

owing table s

nge of the rolhat the palmperpendiculaas viewing th" and "bad" pmages from tware angle mdefault "levlittle finger

han 5 fingersof the fingersft roll was re

shows the re

Pitc

ll of hand pom of the handar to the botthe fingertipspositions wethe digital cmeasuring to

vel" hand posto the thumb

s were held us, whicheveregarded as n

Ro

esults of the v

ch Example

ositions, the d was turned tom of the cas and the angere captured camera of theool. Angles wsition, to a lib if 5 fingersup. If the hanr best represenegative, whi

oll Example

voice-interac

hand was roup) while k

amera imagegle of roll cousing both te last "goodwere measurine segment s were in usend position uented the plaile roll to the

ctive test de

otated left ankeeping the he. The digitaould be clearthe Finger C" positions wred from thewhich ran th

e, or the littlused less thaane of the hae right was r

scribed in E

nd right in a hand parallelal camera warly determine

Counter and twere then e horizontal hrough the pe finger to th

an 4 fingers, and, was useregarded as

xperiment 1

l to as set ed. the

axis, plane he the

ed as

:

Prior to ctesting pothe makeExperimeRAM. Thbe attribularge chubus, usedExperimesecond foExperimeexpect thexperime The folloin the priresponse

Finger Co

Keyboard

conducting Eool. The onlye of the machent 1 was ruhe Pentium Iutable to the unks of data,d for the earlent 1. The fror the earlierent 1. This d

hat the confident and 0.44

owing chart sior experimetime while f

ounter

d

Experiment 1y differencehine runningn on a Dell IIV has a busbus speed o for instancelier experimerame rate of r conducted difference wodence intervfor Experim

shows the avent and Expefive fingers

Figure 1

Mean

3.20s 2.9

2.21s 1.9

1, Stephen C in the testin

g the programInspiron 820s speed of 40on the testinge, images, coent, is three the Finger Cexperiment ould very likal would be

ment 1.

verage respoeriment 1. Inhad the long

. Response T

95% Con

98s

99s

Crampton admng protocols m. The test p00 with an In00 MHz. Theg machines. ould be sent times faster

Counter interand approximkely affect thsimilar, and

onse time by n both experigest.

Times for the

nfidence Int

3.42

2.49

ministered thwere the tes

program on tntel Pentiume difference The bus speto and from than the Pen

rface was apmately 6 frahe mean respd, in fact, it w

finger for thiments, one f

e Finger Cou

terval Range

2s

9s

he same teststing conditithe testing do

m IV 1.4 GHzin results (Sed would afthe CPU. Th

ntium III buspproximatelyames per secponse time. Owas: 0.42s fo

he Finger Cofinger had th

unter

e

t to a differenons, specificone prior to z and 256 M

See Figure 1)ffect how fashe Pentium s, used for y 10 frames pond for One would or the earlier

ounter interfhe shortest

nt cally

MB of ) may st IV

per

r

face

EXPERIFor the d12.6 pixestandard polygonsdotted linand plus

EXPERIFrom theper day rshows th"naturalnCounter w

IMENT 2 drawing test, els for the Fideviations w

s show the mnes connect tshow outlier

IMENT 3 e questionnairanged from e mean and

ness." The mwas the next

Mouse

Finger C

Other Po

the averageinger Countewere 9.8 pixe

middle quartithe nearest srs.

ire response1 to 15, withstandard dev

mouse rankedt "easiest to

Counter

ointing Devi

e distance beer and 6.3 piels and 3.8 piles (25% ansamples with

s, the years oh a spike (6 rviation of thed very high fuse" and sec

ice

tween the drxels for the

pixels. Figurd 75% confi

hin 1.5 times

of computerrespondentse 1-to-10 ran

for both easecond most "n

Ease of Us

Mean

9.0

7.0

6.5

rawing and tother pointin

re 3 shows a idence) surros the inter-qu

r experience ) at 8 hours nkings for "ee of use and "natural."

se "

SD Mea

1.3 9.2

2.2 7.0

2.1 6.1

the circular tng device. Tbox plot of

ounding the uartile range

ranged fromper day. Theease of use" "naturalness

"Naturalness

an SD

0.9

2.5

2.3

template wasThe respectiv

the results. Tmedian, the

es, and the ci

m 6 to 30. Hoe following tand

s," the Finger

s"

D

9

5

3

s ve The ircles

ours table

r

EXPERIThe folloCounter ffinger, wvalue. Thhand pos

IMENT 4 owing graphsfunctions in

which each rehese then detstures, which

s display theregards to o

ed bar displatermined the

h is shown by

e most conseour limitationays as the sme overall rangy the shaded

ervative rangns test. A ran

mallest maximge in which

d gray area.

ge of hand ponge was founmum value athe Finger C

ositions in wnd individuaand the largeCounter func

which the Finally for eachest minimumctioned for a

nger h m all