Research ArticleNonwearable Gaze Tracking System for ControllingHome Appliance

Hwan Heo Jong Man Lee Dongwook Jung Ji Woo Lee and Kang Ryoung Park

Division of Electronics and Electrical Engineering Dongguk University Seoul 100-715 Republic of Korea

Correspondence should be addressed to Kang Ryoung Park parkgrdguedu

Received 10 July 2014 Revised 19 August 2014 Accepted 23 August 2014 Published 14 September 2014

Academic Editor Andrzej Materka

Copyright copy 2014 Hwan Heo et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

A novel gaze tracking system for controlling home appliances in 3D space is proposed in this study Our research is novel in thefollowing four ways First we propose a nonwearable gaze tracking system containing frontal viewing and eye tracking camerasSecond our system includes three modes navigation (for moving the wheelchair depending on the direction of gaze movement)selection (for selecting a specific appliance by gaze estimation) and manipulation (for controlling the selected appliance by gazingat the control panel) The modes can be changed by closing eyes during a specific time period or gazing Third in the navigationmode the signal for moving the wheelchair can be triggered according to the direction of gaze movement Fourth after a specifichome appliance is selected by gazing at it for more than predetermined time period a control panel with 3 times 2 menu is displayedon laptop computer below the gaze tracking system for manipulation The user gazes at one of the menu options for a specific timeperiod which can be manually adjusted according to the user and the signal for controlling the home appliance can be triggeredThe proposed method is shown to have high detection accuracy through a series of experiments

1 Introduction

The rapid development of technology has led to considerableresearch on human-computer interaction to help handi-capped people Mauri et al introduced computer-assistivetechnology for interaction with personal computers viadevices such as switches joysticks trackballs head pointersneural interfaces and eye tracking systems [1] Switchesjoysticks trackballs and head pointers are widely used ascomputer-assistive devices They can be activated by handsfeet chin mouth thumbs palm or head as well as byblowing and can be used as amouse pointer or for controllinga wheelchair [1] Computer vision-based head and facetracking also enables the control of mouse pointers [1ndash3]However thesemethods cannot be easily used by people withsevere disabilities such as quadriplegia a condition in whicha person cannot move hands feet and head

Patients with severe motor disabilities can use differentbioelectrical signals such as electroencephalograms (EEGs)electromyograms (EMGs) and electrooculograms (EOGs)for communication [4] The EEG signal which is based on

cerebral waves can be used to control a virtual keyboardmouse or wheelchair [4 5] Similarly EMG signals whichare based on muscle responses can also be used to interactwith other systems [4 6] The EOG signal can be used forsimple interaction because it determines the approximategaze direction depending on eye movements [4 7 8]However the device that measures EOG signals is expensiveand attaching the EOG sensor around the eye is uncomfort-able Therefore camera-based gaze tracking methods havebeen extensively researched The eye gaze tracking methodsreported previously in [9ndash11] used 2D monitors of desktopcomputers However these methods have some drawbacksFor example the variation in the distance between a userand monitor referred to as 119885-distance henceforth is limitedwhereas different119885-distances exist between a user and differ-ent home appliances in practice In addition the calibrationpositions can be indicated in a monitor in case of thegaze estimation in the 2D monitor However we cannotshow the calibration positions in the monitor because theuser can gaze at the home appliance which does not havemonitor

Hindawi Publishing Corporatione Scientific World JournalVolume 2014 Article ID 303670 20 pageshttpdxdoiorg1011552014303670

2 The Scientific World Journal

To overcome these limitations we propose a novel non-wearable gaze tracking system for controlling home appli-ances in 3D space The proposed system can also be usedby severely disabled people Because our system enables auser to control home appliances by looking at them ratherthan by gazing at a menu on a display it is more naturaland convenient for users This is because the method thatinvolves gazing at a menu must also include a complicatedmenu layout

Existing gaze tracking method based assistive technolo-gies can be categorized as those using a 2D screen and thoseoperating in 3D space The first category technologies usegaze-estimation information to control the cursor on a 2Dscreen [9ndash15] This method uses mapping between the fixedscreen region and the corresponding gaze region In [9 10] awearable eye tracking system on a head-mounted device isused to control a wheelchair and a mouse A nonwearablegaze tracking system based on a visible light web camera isproposed in [11] In this system a face is detected on the basisof skin color and the eye is tracked to control the mouse onthe basis of projections of edge luminance and chrominanceMagee et al proposed an eye gaze tracking method forcontrolling applications such as spelling programs and games[12] An artificial neural network has been used to determinethe relationship between the gaze coordinates and mouseposition thus minimizing mouse jitter caused by saccadiceye movements [13] Lee et al proposed a remote gazedetection method with wide- and narrow-view cameras thathave panning and tilting functionalities and used this systemas an interface for intelligent TVs [14] In [15] a system usinga head-mounted device with two cameras for eye trackingand frontal viewing has been proposed This system can beused by cerebral palsy patients to communicate with othersby selecting symbols on communication boards on a 2Ddisplay

The second category technologies utilize gaze estimationinformation to control an object in 3D space [16ndash21] Shiet al proposed a gaze detection system that contains twocameras for eye tracking and frontal object detection [16]However they used the scale invariant feature transform(SIFT) method to recognize the frontal viewing object thusresulting in a limited 119885-distance range (where the objectcan be recognized) and a high processing time Further therecognition accuracy is degraded if the size of the objectis small with a large 119885-distance between the camera andobject In addition when a TV is turned on different imagesare displayed on the TV screen and they cannot be easilyrecognized by the SIFT algorithm Moreover their methodrequires a user to set up an additional target chart withnine calibration points and gaze at these calibration pointsfor initial calibration which requires considerable processingtime and is inconvenient for the user

In the previous methods [17ndash19] a head-mounted display(HMD) system (of helmet or glasses type) must be wornfor an extended period of time which is inconvenient andtiring In [17] an HMD system with two cameras for eye gazetracking and frontal viewing was proposed for augmentedreality However object recognition in the image capturedby the frontal viewing camera must be performed using

a specific marker pattern In addition wearing the HMDover long periods of time can be inconvenient and may causecybersickness Mardanbegi and Hansen proposed a methodfor users to interact with multiple screens via an HMDcontaining cameras for eye tracking and frontal viewing[18] However their method for identifying the screen inthe frontal viewing image is based on the detection of edgecontours therefore its applicability is limited in cases wherethe screen has a complex background The method in [19]employs an HMD with one camera for eye gaze tracking andtwo scene cameras for frontal viewing By using two scenecameras the viewing range can be increased to match thatof natural head movements of a user However no methodwas proposed for recognition of an object in the scene cameraimage Weibel et al developed a mobile eye tracking systemfor observational research on flight decks [20] However theyused the template matching-basedmethod to identify objectsin images captured by the frontal viewing cameraThis limitsthe 119885-distance range and the viewing angle from which anobject can be recognized because a pilot uses this systemwhile sitting in the cockpit Hales et al proposed a wearabledevice consisting of a camera for eye gaze tracking and onefor object and gesture recognition in the frontal viewingimage [21] However the visible marker based method forobject recognition requires a long processing time and hasa limited 119885-distance range in which the marker can berecognized

Majaranta and Raiha presented the features functional-ities and methods of previous eye typing systems based oneye tracking In addition they proposed a communicationmethod concerned with text production [22] Lankfordpresented the different methods for executing the mouseactions of full ranges and for eye-based typing [23]They usedthe eye gaze response interface computer aid (ERICA) systemby the University of Virginia for experiments Jacob and Karnpresented the different methods of applying eye movementsto user interface (UI) in order to analyze interfaces consider-ing usability and control within a human-computer dialoguePreviously these two issues were dealt with separately JacobandKarn combined them in [24]However [22ndash24] primarilydeal with gaze-based UIs for 2D computer monitors whichis different from the present research of selecting homeappliances in 3D space through gaze tracking

Therefore we propose a new nonwearable gaze trackingsystem for controlling home appliances in 3D space using twonear-infrared (NIR) cameras for gaze tracking and frontalviewing By attaching multiple NIR light-emitting diodes(LEDs) to their outer boundaries different home appliancesin different positions can be easily recognized by the NIRfrontal viewing camera A simple user calibration process inwhich the user gazes at the four corners of a home applianceenables the detection of the final gaze position toward theappliance Table 1 shows the summary of the proposed andexisting methods

The structure of this paper is as follows The proposedsystem and our methodology are presented in Section 2Section 3 describes the experimental setup and the resultsand the conclusions and some ideas for future work arepresented in Section 4

The Scientific World Journal 3

Table 1 Summaries of proposed and previous approaches

Category Method Advantages Disadvantage

Gaze tracking on 2Dscreen

Wearable device [9 10 15]Eye images areacquired by wearablegaze tracking device

The accuracy of gazeestimation is usuallyhigher than that bynonwearable device Application is

restricted tointeraction with a 2DscreenNonwearable device [11ndash14]

Eye or face images areacquired bynonwearable gazetracking device

User convenience ishigher than that bywearable device

Gaze tracking in 3Dspace

Wearable device [16ndash21]Scene and eye imagesare acquired bywearable device

The allowable range ofhead rotation is largebecause the wearablegaze tracking device ismoved according tohead movement

User inconvenienceincreases by wearingthe device includingtwo cameras for gazeestimation and frontalviewing

Nonwearabledevice

Frontal viewingcamera sensingvisible light [16]

Scene and eye imagesare acquired bynonwearable devices

User convenience ishigher than that bywearable device

Complicatedalgorithm is requiredto recognize theobject in the visiblelight image given byfrontal viewingcamera

Frontal viewingcamera sensingNIR light(proposedmethod)

The object in thefrontal viewing imagecan be easily detectedand recognized usingthe NIR camera andNIR LED pattern

Additional NIR LEDsmust be attached tothe outer boundary ofthe object to berecognized

2 Proposed Device and Methods

Figure 1 shows a photograph of the proposed device withthe eye tracking and scene cameras and an NIR illuminatorThese are the nonwearable devices attached to the wheelchairA Logitech C600 commercial web camera with a universalserial bus (USB) interface is used for the eye tracking andscene cameras [25] The NIR cutting filter inside the camerais replaced by an NIR passing filter to ensure that the imagescaptured by the eye tracking and scene cameras are notaffected by the exterior lighting conditions The accuracy ofgaze estimation is typically higher with high-resolution eyeimages than that with low-resolution eye images Thereforethe eye tracking camera has a resolution of 1600times1200 pixelsBecause the accuracy of detecting NIR LED spots in a sceneimage is minimally affected by image resolution the scenecamera produces images of 640 times 480 pixels consideringthe bandwidth limit of data transfer in a USB 20 webcamera Considering the synchronization of images of theeye tracking and scene cameras we use images acquired atapproximately 10 fps for our system The eye camera is posi-tioned below the eye to prevent obstruction of line of sightof the user at a119885-distance of approximately 55 cmndash60 cm Toobtain a large eye image at this 119885-distance the eye camerais equipped with a zoom lens The illuminator of 8 times 8NIRLEDs is attached below the eye tracking camera as shown inFigure 1 The wavelength of the NIR LEDs is approximately850 nmwhich does not dazzle the eye of the user but provides

a distinctive boundary between the pupil and the iris Thedotted circles in Figure 1 indicate the NIR LEDs

Figure 2 shows the operation flowchart of the proposedsystem After the system is turned on the nonwearable eyetracking and scene cameras on the wheelchair capture theeye and scene images respectively In the eye image thecenters of the corneal specular reflection (SR) and the pupilare detected as explained in Section 21 In the scene imagethe spots of the NIR LEDs are located as explained inSection 22 In the initial calibration stage the user gazes atthe four corners of the home appliance to obtain themappingmatrix between the pupilrsquos movable region in the eye imageand the object region in the scene image (Section 23) Thehome appliance is recognized by the scene camera on thebasis of the number of NIR LED spots and their patterns(Section 22)The userrsquos gaze position is then calculated usingthe mapping matrix in the scene image (Section 24) If thegaze position is located in the region of a home appliance inthe scene image the system selects that home appliance forcontrol Even if the gaze position is located at the corner oron the boundary of the home appliance region in the sceneimage the proposed system selects the home appliance to becontrolled

21 Detecting Corneal SR and Pupil After an image iscaptured by the eye tracking camera the corneal SR and pupilare detected as shown in Figure 3

4 The Scientific World Journal

Eye tracking camera

Scene camera

NIR LED illuminator

About 270 or 300 cmAbout 55sim60cm

Figure 1 Proposed system

Starting proposedsystem

Capturing eye image Capturing scene image

Detecting the centers ofcorneal SR and pupil

Detecting the spots ofNIR LEDs

Calibrationstage

Recognizing the homeappliance by using scene

camera

Gazing at four cornerpositions of home

appliance

Yes

No

Calculating gaze positionin the scene image

Does gaze positionexist in the region ofthe home appliance

Selecting the homeappliance for controlling

Yes

No

Figure 2 Flowchart of the proposed system

Figure 3 shows the flowchart for the detection of thecorneal SR and pupil regions After the eye tracking cameracaptures an image the candidate corneal SRs in a prede-fined search region are extracted using image binarizationcomponent labeling and size filtering [26] The regions ofinterest (ROIs) for detecting the pupil are then defineddepending on the detected corneal SR candidates andthe approximate position of the pupil is determined usingsubblock-based matching In the subblock-based matchingprocess nine subblocks are defined and the position at which

the difference between the mean of the central subblock (4in Figure 4) and the surrounding subblocks is maximizedis determined to be the pupil region [27] This procedureis repeated by moving the 3 times 3 mask with these ninesubblocks in the ROIs The integral imaging method is usedto reduce the computational complexity by calculating theaverage intensity of each subblock [27]

If no corneal SR is detected in the search region thesubblock-based matching process is performed in this areaThe subblock-based matching method uses different sub-block sizes (from 20 times 20 pixels to 60 times 60 pixels) dependingon the pupil size which is affected by variations in illumi-nation and the 119885-distance between the camera and userrsquoseye Figure 5 shows an example of detecting the corneal SRand approximate pupil regions Figure 6 shows the result ofsample pupil detection

The pupil region shown in Figure 6 is the approximateposition of the pupil An accurate estimate of the center of thepupil is required to accurately determine the gaze positionFor detecting the accurate pupil center an ROI is definedon the basis of the center of the approximate pupil regionas shown in Figure 6 Within this ROI the pupil center isaccurately located using the procedure shown in Figure 7

Figure 7 shows the flowchart of the accurate pupil centerdetection method First an image binarization is performedon the basis of the ROI shown in Figure 6 as shownin Figure 8(b) The threshold value for image binarizationis determined using Gonzalezrsquos method [28] Canny edgedetection is performed to locate the edge of the pupil region[29] as shown inFigure 8(c)Thepupil center is thendetectedwith an ellipse fitting algorithm as shown in Figure 8(d)The final result of the pupil center detection is shown inFigure 8(e)

The primary differences between the proposed and theother video-based combined pupilcorneal reflection meth-ods are as follows In our research the ROI for eye detection issignificantly reduced by the subblock-based template match-ing algorithm within the left (or right) corneal SR searchregion (as shown in Figure 5) for high accuracy and fastprocessing of eye detection In addition the accurate pupilcenter is located via ellipse fitting as shown in Figure 7

22 Recognizing Home Appliances Using Frontal ViewingCamera In the proposed system home appliances such asTVs heaters and air conditioner switches are recognized bythe scene camera shown in Figure 1 In the images captured bythe scene camera the shape of the home appliances may varydepending on the position of the scene camera In additionthey may also change with illumination variations If theTV is turned on different shapes may be produced by theprogram running on the TV Therefore recognition of everyhome appliance is very difficult To overcome these problemsmultipleNIR LEDs are attached at the outer boundaries of thehome appliances (the red dotted circles of Figures 9(b)ndash9(d))and an NIR scene camera is employed The camera is robustto different conditions such as variations in exterior lightingconditions object size and viewpointsThewavelength of theNIR LEDs is 850 nm

The Scientific World Journal 5

(1) Capturing eye image

(2) Extracting the candidates ofcorneal SR in search region

(3) Is there corneal SRin search region

(4) Defining regions of interest (ROIs)based on the detected corneal SRs

(6) Detecting pupil region in the ROIsby using subblock-based matching

(7) Selecting the area where thematching score is highest

as a pupil region

(8) Locating accurate pupil centerby ellipse fitting

(9) Selecting the region whose centeris closest to the center of pupil

as a corneal SR

(5) Detecting pupil region in thesearch region of corneal SR

by using subblock-based matching

No

Yes

Figure 3 Flowchart for detection of corneal SR and pupil regions

0 1 2

3 4 5

6 7 8

Figure 4 Example of subblock-based matching for pupil detection

To allow user-dependent calibration for eye tracking(Section 23) four NIR LEDs are attached at the four cornersof home appliances as shown in Figures 9(b)ndash9(d) Becausethe air conditioner switch is too small to be recognized bythe scene camera the NIR LEDs are attached at positionsslightly outside the four corners of the switch as shown inFigure 9(d)

Therefore the home appliances can be recognized onbasis of the patterns and the number of NIR LEDs detected

Subblock-based matchingusing 3 times 3 window mask (steps 5 and 6 of Figure 3)

Detected corneal SRfrom step 2 of Figure 3

ROI determinedon the basis of corneal SR

(step 4 of Figure 3)If the corneal SR is detected

in the search regionIf the corneal SR is not detected

in the search region

The search region ofleft corneal SR

for step 2 of Figure 3

The search region ofright corneal SR

for step 2 of Figure 3

Figure 5 Sample detection of the corneal SR and approximate pupilregions

by the NIR scene camera as shown in Figure 10 The spec-ifications of the NIR LEDs and NIR camera indicate thatonly the spots of the NIR-LEDs can be seen in the cameraimage Thereafter the NIR LEDs in the image captured bythe scene camera can be easily extracted by binarization

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

2 The Scientific World Journal

To overcome these limitations we propose a novel non-wearable gaze tracking system for controlling home appli-ances in 3D space The proposed system can also be usedby severely disabled people Because our system enables auser to control home appliances by looking at them ratherthan by gazing at a menu on a display it is more naturaland convenient for users This is because the method thatinvolves gazing at a menu must also include a complicatedmenu layout

Existing gaze tracking method based assistive technolo-gies can be categorized as those using a 2D screen and thoseoperating in 3D space The first category technologies usegaze-estimation information to control the cursor on a 2Dscreen [9ndash15] This method uses mapping between the fixedscreen region and the corresponding gaze region In [9 10] awearable eye tracking system on a head-mounted device isused to control a wheelchair and a mouse A nonwearablegaze tracking system based on a visible light web camera isproposed in [11] In this system a face is detected on the basisof skin color and the eye is tracked to control the mouse onthe basis of projections of edge luminance and chrominanceMagee et al proposed an eye gaze tracking method forcontrolling applications such as spelling programs and games[12] An artificial neural network has been used to determinethe relationship between the gaze coordinates and mouseposition thus minimizing mouse jitter caused by saccadiceye movements [13] Lee et al proposed a remote gazedetection method with wide- and narrow-view cameras thathave panning and tilting functionalities and used this systemas an interface for intelligent TVs [14] In [15] a system usinga head-mounted device with two cameras for eye trackingand frontal viewing has been proposed This system can beused by cerebral palsy patients to communicate with othersby selecting symbols on communication boards on a 2Ddisplay

The second category technologies utilize gaze estimationinformation to control an object in 3D space [16ndash21] Shiet al proposed a gaze detection system that contains twocameras for eye tracking and frontal object detection [16]However they used the scale invariant feature transform(SIFT) method to recognize the frontal viewing object thusresulting in a limited 119885-distance range (where the objectcan be recognized) and a high processing time Further therecognition accuracy is degraded if the size of the objectis small with a large 119885-distance between the camera andobject In addition when a TV is turned on different imagesare displayed on the TV screen and they cannot be easilyrecognized by the SIFT algorithm Moreover their methodrequires a user to set up an additional target chart withnine calibration points and gaze at these calibration pointsfor initial calibration which requires considerable processingtime and is inconvenient for the user

In the previous methods [17ndash19] a head-mounted display(HMD) system (of helmet or glasses type) must be wornfor an extended period of time which is inconvenient andtiring In [17] an HMD system with two cameras for eye gazetracking and frontal viewing was proposed for augmentedreality However object recognition in the image capturedby the frontal viewing camera must be performed using

a specific marker pattern In addition wearing the HMDover long periods of time can be inconvenient and may causecybersickness Mardanbegi and Hansen proposed a methodfor users to interact with multiple screens via an HMDcontaining cameras for eye tracking and frontal viewing[18] However their method for identifying the screen inthe frontal viewing image is based on the detection of edgecontours therefore its applicability is limited in cases wherethe screen has a complex background The method in [19]employs an HMD with one camera for eye gaze tracking andtwo scene cameras for frontal viewing By using two scenecameras the viewing range can be increased to match thatof natural head movements of a user However no methodwas proposed for recognition of an object in the scene cameraimage Weibel et al developed a mobile eye tracking systemfor observational research on flight decks [20] However theyused the template matching-basedmethod to identify objectsin images captured by the frontal viewing cameraThis limitsthe 119885-distance range and the viewing angle from which anobject can be recognized because a pilot uses this systemwhile sitting in the cockpit Hales et al proposed a wearabledevice consisting of a camera for eye gaze tracking and onefor object and gesture recognition in the frontal viewingimage [21] However the visible marker based method forobject recognition requires a long processing time and hasa limited 119885-distance range in which the marker can berecognized

Majaranta and Raiha presented the features functional-ities and methods of previous eye typing systems based oneye tracking In addition they proposed a communicationmethod concerned with text production [22] Lankfordpresented the different methods for executing the mouseactions of full ranges and for eye-based typing [23]They usedthe eye gaze response interface computer aid (ERICA) systemby the University of Virginia for experiments Jacob and Karnpresented the different methods of applying eye movementsto user interface (UI) in order to analyze interfaces consider-ing usability and control within a human-computer dialoguePreviously these two issues were dealt with separately JacobandKarn combined them in [24]However [22ndash24] primarilydeal with gaze-based UIs for 2D computer monitors whichis different from the present research of selecting homeappliances in 3D space through gaze tracking

Therefore we propose a new nonwearable gaze trackingsystem for controlling home appliances in 3D space using twonear-infrared (NIR) cameras for gaze tracking and frontalviewing By attaching multiple NIR light-emitting diodes(LEDs) to their outer boundaries different home appliancesin different positions can be easily recognized by the NIRfrontal viewing camera A simple user calibration process inwhich the user gazes at the four corners of a home applianceenables the detection of the final gaze position toward theappliance Table 1 shows the summary of the proposed andexisting methods

The structure of this paper is as follows The proposedsystem and our methodology are presented in Section 2Section 3 describes the experimental setup and the resultsand the conclusions and some ideas for future work arepresented in Section 4

The Scientific World Journal 3

Table 1 Summaries of proposed and previous approaches

Category Method Advantages Disadvantage

Gaze tracking on 2Dscreen

Wearable device [9 10 15]Eye images areacquired by wearablegaze tracking device

The accuracy of gazeestimation is usuallyhigher than that bynonwearable device Application is

restricted tointeraction with a 2DscreenNonwearable device [11ndash14]

Eye or face images areacquired bynonwearable gazetracking device

User convenience ishigher than that bywearable device

Gaze tracking in 3Dspace

Wearable device [16ndash21]Scene and eye imagesare acquired bywearable device

The allowable range ofhead rotation is largebecause the wearablegaze tracking device ismoved according tohead movement

User inconvenienceincreases by wearingthe device includingtwo cameras for gazeestimation and frontalviewing

Nonwearabledevice

Frontal viewingcamera sensingvisible light [16]

Scene and eye imagesare acquired bynonwearable devices

User convenience ishigher than that bywearable device

Complicatedalgorithm is requiredto recognize theobject in the visiblelight image given byfrontal viewingcamera

Frontal viewingcamera sensingNIR light(proposedmethod)

The object in thefrontal viewing imagecan be easily detectedand recognized usingthe NIR camera andNIR LED pattern

Additional NIR LEDsmust be attached tothe outer boundary ofthe object to berecognized

2 Proposed Device and Methods

Figure 1 shows a photograph of the proposed device withthe eye tracking and scene cameras and an NIR illuminatorThese are the nonwearable devices attached to the wheelchairA Logitech C600 commercial web camera with a universalserial bus (USB) interface is used for the eye tracking andscene cameras [25] The NIR cutting filter inside the camerais replaced by an NIR passing filter to ensure that the imagescaptured by the eye tracking and scene cameras are notaffected by the exterior lighting conditions The accuracy ofgaze estimation is typically higher with high-resolution eyeimages than that with low-resolution eye images Thereforethe eye tracking camera has a resolution of 1600times1200 pixelsBecause the accuracy of detecting NIR LED spots in a sceneimage is minimally affected by image resolution the scenecamera produces images of 640 times 480 pixels consideringthe bandwidth limit of data transfer in a USB 20 webcamera Considering the synchronization of images of theeye tracking and scene cameras we use images acquired atapproximately 10 fps for our system The eye camera is posi-tioned below the eye to prevent obstruction of line of sightof the user at a119885-distance of approximately 55 cmndash60 cm Toobtain a large eye image at this 119885-distance the eye camerais equipped with a zoom lens The illuminator of 8 times 8NIRLEDs is attached below the eye tracking camera as shown inFigure 1 The wavelength of the NIR LEDs is approximately850 nmwhich does not dazzle the eye of the user but provides

a distinctive boundary between the pupil and the iris Thedotted circles in Figure 1 indicate the NIR LEDs

Figure 2 shows the operation flowchart of the proposedsystem After the system is turned on the nonwearable eyetracking and scene cameras on the wheelchair capture theeye and scene images respectively In the eye image thecenters of the corneal specular reflection (SR) and the pupilare detected as explained in Section 21 In the scene imagethe spots of the NIR LEDs are located as explained inSection 22 In the initial calibration stage the user gazes atthe four corners of the home appliance to obtain themappingmatrix between the pupilrsquos movable region in the eye imageand the object region in the scene image (Section 23) Thehome appliance is recognized by the scene camera on thebasis of the number of NIR LED spots and their patterns(Section 22)The userrsquos gaze position is then calculated usingthe mapping matrix in the scene image (Section 24) If thegaze position is located in the region of a home appliance inthe scene image the system selects that home appliance forcontrol Even if the gaze position is located at the corner oron the boundary of the home appliance region in the sceneimage the proposed system selects the home appliance to becontrolled

21 Detecting Corneal SR and Pupil After an image iscaptured by the eye tracking camera the corneal SR and pupilare detected as shown in Figure 3

4 The Scientific World Journal

Eye tracking camera

Scene camera

NIR LED illuminator

About 270 or 300 cmAbout 55sim60cm

Figure 1 Proposed system

Starting proposedsystem

Capturing eye image Capturing scene image

Detecting the centers ofcorneal SR and pupil

Detecting the spots ofNIR LEDs

Calibrationstage

Recognizing the homeappliance by using scene

camera

Gazing at four cornerpositions of home

appliance

Yes

No

Calculating gaze positionin the scene image

Does gaze positionexist in the region ofthe home appliance

Selecting the homeappliance for controlling

Yes

No

Figure 2 Flowchart of the proposed system

Figure 3 shows the flowchart for the detection of thecorneal SR and pupil regions After the eye tracking cameracaptures an image the candidate corneal SRs in a prede-fined search region are extracted using image binarizationcomponent labeling and size filtering [26] The regions ofinterest (ROIs) for detecting the pupil are then defineddepending on the detected corneal SR candidates andthe approximate position of the pupil is determined usingsubblock-based matching In the subblock-based matchingprocess nine subblocks are defined and the position at which

the difference between the mean of the central subblock (4in Figure 4) and the surrounding subblocks is maximizedis determined to be the pupil region [27] This procedureis repeated by moving the 3 times 3 mask with these ninesubblocks in the ROIs The integral imaging method is usedto reduce the computational complexity by calculating theaverage intensity of each subblock [27]

If no corneal SR is detected in the search region thesubblock-based matching process is performed in this areaThe subblock-based matching method uses different sub-block sizes (from 20 times 20 pixels to 60 times 60 pixels) dependingon the pupil size which is affected by variations in illumi-nation and the 119885-distance between the camera and userrsquoseye Figure 5 shows an example of detecting the corneal SRand approximate pupil regions Figure 6 shows the result ofsample pupil detection

The pupil region shown in Figure 6 is the approximateposition of the pupil An accurate estimate of the center of thepupil is required to accurately determine the gaze positionFor detecting the accurate pupil center an ROI is definedon the basis of the center of the approximate pupil regionas shown in Figure 6 Within this ROI the pupil center isaccurately located using the procedure shown in Figure 7

Figure 7 shows the flowchart of the accurate pupil centerdetection method First an image binarization is performedon the basis of the ROI shown in Figure 6 as shownin Figure 8(b) The threshold value for image binarizationis determined using Gonzalezrsquos method [28] Canny edgedetection is performed to locate the edge of the pupil region[29] as shown inFigure 8(c)Thepupil center is thendetectedwith an ellipse fitting algorithm as shown in Figure 8(d)The final result of the pupil center detection is shown inFigure 8(e)

The primary differences between the proposed and theother video-based combined pupilcorneal reflection meth-ods are as follows In our research the ROI for eye detection issignificantly reduced by the subblock-based template match-ing algorithm within the left (or right) corneal SR searchregion (as shown in Figure 5) for high accuracy and fastprocessing of eye detection In addition the accurate pupilcenter is located via ellipse fitting as shown in Figure 7

22 Recognizing Home Appliances Using Frontal ViewingCamera In the proposed system home appliances such asTVs heaters and air conditioner switches are recognized bythe scene camera shown in Figure 1 In the images captured bythe scene camera the shape of the home appliances may varydepending on the position of the scene camera In additionthey may also change with illumination variations If theTV is turned on different shapes may be produced by theprogram running on the TV Therefore recognition of everyhome appliance is very difficult To overcome these problemsmultipleNIR LEDs are attached at the outer boundaries of thehome appliances (the red dotted circles of Figures 9(b)ndash9(d))and an NIR scene camera is employed The camera is robustto different conditions such as variations in exterior lightingconditions object size and viewpointsThewavelength of theNIR LEDs is 850 nm

The Scientific World Journal 5

(1) Capturing eye image

(2) Extracting the candidates ofcorneal SR in search region

(3) Is there corneal SRin search region

(4) Defining regions of interest (ROIs)based on the detected corneal SRs

(6) Detecting pupil region in the ROIsby using subblock-based matching

(7) Selecting the area where thematching score is highest

as a pupil region

(8) Locating accurate pupil centerby ellipse fitting

(9) Selecting the region whose centeris closest to the center of pupil

as a corneal SR

(5) Detecting pupil region in thesearch region of corneal SR

by using subblock-based matching

No

Yes

Figure 3 Flowchart for detection of corneal SR and pupil regions

0 1 2

3 4 5

6 7 8

Figure 4 Example of subblock-based matching for pupil detection

To allow user-dependent calibration for eye tracking(Section 23) four NIR LEDs are attached at the four cornersof home appliances as shown in Figures 9(b)ndash9(d) Becausethe air conditioner switch is too small to be recognized bythe scene camera the NIR LEDs are attached at positionsslightly outside the four corners of the switch as shown inFigure 9(d)

Therefore the home appliances can be recognized onbasis of the patterns and the number of NIR LEDs detected

Subblock-based matchingusing 3 times 3 window mask (steps 5 and 6 of Figure 3)

Detected corneal SRfrom step 2 of Figure 3

ROI determinedon the basis of corneal SR

(step 4 of Figure 3)If the corneal SR is detected

in the search regionIf the corneal SR is not detected

in the search region

The search region ofleft corneal SR

for step 2 of Figure 3

The search region ofright corneal SR

for step 2 of Figure 3

Figure 5 Sample detection of the corneal SR and approximate pupilregions

by the NIR scene camera as shown in Figure 10 The spec-ifications of the NIR LEDs and NIR camera indicate thatonly the spots of the NIR-LEDs can be seen in the cameraimage Thereafter the NIR LEDs in the image captured bythe scene camera can be easily extracted by binarization

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 3

Table 1 Summaries of proposed and previous approaches

Category Method Advantages Disadvantage

Gaze tracking on 2Dscreen

Wearable device [9 10 15]Eye images areacquired by wearablegaze tracking device

The accuracy of gazeestimation is usuallyhigher than that bynonwearable device Application is

restricted tointeraction with a 2DscreenNonwearable device [11ndash14]

Eye or face images areacquired bynonwearable gazetracking device

User convenience ishigher than that bywearable device

Gaze tracking in 3Dspace

Wearable device [16ndash21]Scene and eye imagesare acquired bywearable device

The allowable range ofhead rotation is largebecause the wearablegaze tracking device ismoved according tohead movement

User inconvenienceincreases by wearingthe device includingtwo cameras for gazeestimation and frontalviewing

Nonwearabledevice

Frontal viewingcamera sensingvisible light [16]

Scene and eye imagesare acquired bynonwearable devices

User convenience ishigher than that bywearable device

Complicatedalgorithm is requiredto recognize theobject in the visiblelight image given byfrontal viewingcamera

Frontal viewingcamera sensingNIR light(proposedmethod)

The object in thefrontal viewing imagecan be easily detectedand recognized usingthe NIR camera andNIR LED pattern

Additional NIR LEDsmust be attached tothe outer boundary ofthe object to berecognized

2 Proposed Device and Methods

Figure 1 shows a photograph of the proposed device withthe eye tracking and scene cameras and an NIR illuminatorThese are the nonwearable devices attached to the wheelchairA Logitech C600 commercial web camera with a universalserial bus (USB) interface is used for the eye tracking andscene cameras [25] The NIR cutting filter inside the camerais replaced by an NIR passing filter to ensure that the imagescaptured by the eye tracking and scene cameras are notaffected by the exterior lighting conditions The accuracy ofgaze estimation is typically higher with high-resolution eyeimages than that with low-resolution eye images Thereforethe eye tracking camera has a resolution of 1600times1200 pixelsBecause the accuracy of detecting NIR LED spots in a sceneimage is minimally affected by image resolution the scenecamera produces images of 640 times 480 pixels consideringthe bandwidth limit of data transfer in a USB 20 webcamera Considering the synchronization of images of theeye tracking and scene cameras we use images acquired atapproximately 10 fps for our system The eye camera is posi-tioned below the eye to prevent obstruction of line of sightof the user at a119885-distance of approximately 55 cmndash60 cm Toobtain a large eye image at this 119885-distance the eye camerais equipped with a zoom lens The illuminator of 8 times 8NIRLEDs is attached below the eye tracking camera as shown inFigure 1 The wavelength of the NIR LEDs is approximately850 nmwhich does not dazzle the eye of the user but provides

a distinctive boundary between the pupil and the iris Thedotted circles in Figure 1 indicate the NIR LEDs

Figure 2 shows the operation flowchart of the proposedsystem After the system is turned on the nonwearable eyetracking and scene cameras on the wheelchair capture theeye and scene images respectively In the eye image thecenters of the corneal specular reflection (SR) and the pupilare detected as explained in Section 21 In the scene imagethe spots of the NIR LEDs are located as explained inSection 22 In the initial calibration stage the user gazes atthe four corners of the home appliance to obtain themappingmatrix between the pupilrsquos movable region in the eye imageand the object region in the scene image (Section 23) Thehome appliance is recognized by the scene camera on thebasis of the number of NIR LED spots and their patterns(Section 22)The userrsquos gaze position is then calculated usingthe mapping matrix in the scene image (Section 24) If thegaze position is located in the region of a home appliance inthe scene image the system selects that home appliance forcontrol Even if the gaze position is located at the corner oron the boundary of the home appliance region in the sceneimage the proposed system selects the home appliance to becontrolled

21 Detecting Corneal SR and Pupil After an image iscaptured by the eye tracking camera the corneal SR and pupilare detected as shown in Figure 3

4 The Scientific World Journal

Eye tracking camera

Scene camera

NIR LED illuminator

About 270 or 300 cmAbout 55sim60cm

Figure 1 Proposed system

Starting proposedsystem

Capturing eye image Capturing scene image

Detecting the centers ofcorneal SR and pupil

Detecting the spots ofNIR LEDs

Calibrationstage

Recognizing the homeappliance by using scene

camera

Gazing at four cornerpositions of home

appliance

Yes

No

Calculating gaze positionin the scene image

Does gaze positionexist in the region ofthe home appliance

Selecting the homeappliance for controlling

Yes

No

Figure 2 Flowchart of the proposed system

Figure 3 shows the flowchart for the detection of thecorneal SR and pupil regions After the eye tracking cameracaptures an image the candidate corneal SRs in a prede-fined search region are extracted using image binarizationcomponent labeling and size filtering [26] The regions ofinterest (ROIs) for detecting the pupil are then defineddepending on the detected corneal SR candidates andthe approximate position of the pupil is determined usingsubblock-based matching In the subblock-based matchingprocess nine subblocks are defined and the position at which

the difference between the mean of the central subblock (4in Figure 4) and the surrounding subblocks is maximizedis determined to be the pupil region [27] This procedureis repeated by moving the 3 times 3 mask with these ninesubblocks in the ROIs The integral imaging method is usedto reduce the computational complexity by calculating theaverage intensity of each subblock [27]

If no corneal SR is detected in the search region thesubblock-based matching process is performed in this areaThe subblock-based matching method uses different sub-block sizes (from 20 times 20 pixels to 60 times 60 pixels) dependingon the pupil size which is affected by variations in illumi-nation and the 119885-distance between the camera and userrsquoseye Figure 5 shows an example of detecting the corneal SRand approximate pupil regions Figure 6 shows the result ofsample pupil detection

The pupil region shown in Figure 6 is the approximateposition of the pupil An accurate estimate of the center of thepupil is required to accurately determine the gaze positionFor detecting the accurate pupil center an ROI is definedon the basis of the center of the approximate pupil regionas shown in Figure 6 Within this ROI the pupil center isaccurately located using the procedure shown in Figure 7

Figure 7 shows the flowchart of the accurate pupil centerdetection method First an image binarization is performedon the basis of the ROI shown in Figure 6 as shownin Figure 8(b) The threshold value for image binarizationis determined using Gonzalezrsquos method [28] Canny edgedetection is performed to locate the edge of the pupil region[29] as shown inFigure 8(c)Thepupil center is thendetectedwith an ellipse fitting algorithm as shown in Figure 8(d)The final result of the pupil center detection is shown inFigure 8(e)

The primary differences between the proposed and theother video-based combined pupilcorneal reflection meth-ods are as follows In our research the ROI for eye detection issignificantly reduced by the subblock-based template match-ing algorithm within the left (or right) corneal SR searchregion (as shown in Figure 5) for high accuracy and fastprocessing of eye detection In addition the accurate pupilcenter is located via ellipse fitting as shown in Figure 7

22 Recognizing Home Appliances Using Frontal ViewingCamera In the proposed system home appliances such asTVs heaters and air conditioner switches are recognized bythe scene camera shown in Figure 1 In the images captured bythe scene camera the shape of the home appliances may varydepending on the position of the scene camera In additionthey may also change with illumination variations If theTV is turned on different shapes may be produced by theprogram running on the TV Therefore recognition of everyhome appliance is very difficult To overcome these problemsmultipleNIR LEDs are attached at the outer boundaries of thehome appliances (the red dotted circles of Figures 9(b)ndash9(d))and an NIR scene camera is employed The camera is robustto different conditions such as variations in exterior lightingconditions object size and viewpointsThewavelength of theNIR LEDs is 850 nm

The Scientific World Journal 5

(1) Capturing eye image

(2) Extracting the candidates ofcorneal SR in search region

(3) Is there corneal SRin search region

(4) Defining regions of interest (ROIs)based on the detected corneal SRs

(6) Detecting pupil region in the ROIsby using subblock-based matching

(7) Selecting the area where thematching score is highest

as a pupil region

(8) Locating accurate pupil centerby ellipse fitting

(9) Selecting the region whose centeris closest to the center of pupil

as a corneal SR

(5) Detecting pupil region in thesearch region of corneal SR

by using subblock-based matching

No

Yes

Figure 3 Flowchart for detection of corneal SR and pupil regions

0 1 2

3 4 5

6 7 8

Figure 4 Example of subblock-based matching for pupil detection

To allow user-dependent calibration for eye tracking(Section 23) four NIR LEDs are attached at the four cornersof home appliances as shown in Figures 9(b)ndash9(d) Becausethe air conditioner switch is too small to be recognized bythe scene camera the NIR LEDs are attached at positionsslightly outside the four corners of the switch as shown inFigure 9(d)

Therefore the home appliances can be recognized onbasis of the patterns and the number of NIR LEDs detected

Subblock-based matchingusing 3 times 3 window mask (steps 5 and 6 of Figure 3)

Detected corneal SRfrom step 2 of Figure 3

ROI determinedon the basis of corneal SR

(step 4 of Figure 3)If the corneal SR is detected

in the search regionIf the corneal SR is not detected

in the search region

The search region ofleft corneal SR

for step 2 of Figure 3

The search region ofright corneal SR

for step 2 of Figure 3

Figure 5 Sample detection of the corneal SR and approximate pupilregions

by the NIR scene camera as shown in Figure 10 The spec-ifications of the NIR LEDs and NIR camera indicate thatonly the spots of the NIR-LEDs can be seen in the cameraimage Thereafter the NIR LEDs in the image captured bythe scene camera can be easily extracted by binarization

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

4 The Scientific World Journal

Eye tracking camera

Scene camera

NIR LED illuminator

About 270 or 300 cmAbout 55sim60cm

Figure 1 Proposed system

Starting proposedsystem

Capturing eye image Capturing scene image

Detecting the centers ofcorneal SR and pupil

Detecting the spots ofNIR LEDs

Calibrationstage

Recognizing the homeappliance by using scene

camera

Gazing at four cornerpositions of home

appliance

Yes

No

Calculating gaze positionin the scene image

Does gaze positionexist in the region ofthe home appliance

Selecting the homeappliance for controlling

Yes

No

Figure 2 Flowchart of the proposed system

Figure 3 shows the flowchart for the detection of thecorneal SR and pupil regions After the eye tracking cameracaptures an image the candidate corneal SRs in a prede-fined search region are extracted using image binarizationcomponent labeling and size filtering [26] The regions ofinterest (ROIs) for detecting the pupil are then defineddepending on the detected corneal SR candidates andthe approximate position of the pupil is determined usingsubblock-based matching In the subblock-based matchingprocess nine subblocks are defined and the position at which

the difference between the mean of the central subblock (4in Figure 4) and the surrounding subblocks is maximizedis determined to be the pupil region [27] This procedureis repeated by moving the 3 times 3 mask with these ninesubblocks in the ROIs The integral imaging method is usedto reduce the computational complexity by calculating theaverage intensity of each subblock [27]

If no corneal SR is detected in the search region thesubblock-based matching process is performed in this areaThe subblock-based matching method uses different sub-block sizes (from 20 times 20 pixels to 60 times 60 pixels) dependingon the pupil size which is affected by variations in illumi-nation and the 119885-distance between the camera and userrsquoseye Figure 5 shows an example of detecting the corneal SRand approximate pupil regions Figure 6 shows the result ofsample pupil detection

The pupil region shown in Figure 6 is the approximateposition of the pupil An accurate estimate of the center of thepupil is required to accurately determine the gaze positionFor detecting the accurate pupil center an ROI is definedon the basis of the center of the approximate pupil regionas shown in Figure 6 Within this ROI the pupil center isaccurately located using the procedure shown in Figure 7

Figure 7 shows the flowchart of the accurate pupil centerdetection method First an image binarization is performedon the basis of the ROI shown in Figure 6 as shownin Figure 8(b) The threshold value for image binarizationis determined using Gonzalezrsquos method [28] Canny edgedetection is performed to locate the edge of the pupil region[29] as shown inFigure 8(c)Thepupil center is thendetectedwith an ellipse fitting algorithm as shown in Figure 8(d)The final result of the pupil center detection is shown inFigure 8(e)

The primary differences between the proposed and theother video-based combined pupilcorneal reflection meth-ods are as follows In our research the ROI for eye detection issignificantly reduced by the subblock-based template match-ing algorithm within the left (or right) corneal SR searchregion (as shown in Figure 5) for high accuracy and fastprocessing of eye detection In addition the accurate pupilcenter is located via ellipse fitting as shown in Figure 7

22 Recognizing Home Appliances Using Frontal ViewingCamera In the proposed system home appliances such asTVs heaters and air conditioner switches are recognized bythe scene camera shown in Figure 1 In the images captured bythe scene camera the shape of the home appliances may varydepending on the position of the scene camera In additionthey may also change with illumination variations If theTV is turned on different shapes may be produced by theprogram running on the TV Therefore recognition of everyhome appliance is very difficult To overcome these problemsmultipleNIR LEDs are attached at the outer boundaries of thehome appliances (the red dotted circles of Figures 9(b)ndash9(d))and an NIR scene camera is employed The camera is robustto different conditions such as variations in exterior lightingconditions object size and viewpointsThewavelength of theNIR LEDs is 850 nm

The Scientific World Journal 5

(1) Capturing eye image

(2) Extracting the candidates ofcorneal SR in search region

(3) Is there corneal SRin search region

(4) Defining regions of interest (ROIs)based on the detected corneal SRs

(6) Detecting pupil region in the ROIsby using subblock-based matching

(7) Selecting the area where thematching score is highest

as a pupil region

(8) Locating accurate pupil centerby ellipse fitting

(9) Selecting the region whose centeris closest to the center of pupil

as a corneal SR

(5) Detecting pupil region in thesearch region of corneal SR

by using subblock-based matching

No

Yes

Figure 3 Flowchart for detection of corneal SR and pupil regions

0 1 2

3 4 5

6 7 8

Figure 4 Example of subblock-based matching for pupil detection

To allow user-dependent calibration for eye tracking(Section 23) four NIR LEDs are attached at the four cornersof home appliances as shown in Figures 9(b)ndash9(d) Becausethe air conditioner switch is too small to be recognized bythe scene camera the NIR LEDs are attached at positionsslightly outside the four corners of the switch as shown inFigure 9(d)

Therefore the home appliances can be recognized onbasis of the patterns and the number of NIR LEDs detected

Subblock-based matchingusing 3 times 3 window mask (steps 5 and 6 of Figure 3)

Detected corneal SRfrom step 2 of Figure 3

ROI determinedon the basis of corneal SR

(step 4 of Figure 3)If the corneal SR is detected

in the search regionIf the corneal SR is not detected

in the search region

The search region ofleft corneal SR

for step 2 of Figure 3

The search region ofright corneal SR

for step 2 of Figure 3

Figure 5 Sample detection of the corneal SR and approximate pupilregions

by the NIR scene camera as shown in Figure 10 The spec-ifications of the NIR LEDs and NIR camera indicate thatonly the spots of the NIR-LEDs can be seen in the cameraimage Thereafter the NIR LEDs in the image captured bythe scene camera can be easily extracted by binarization

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 5

(1) Capturing eye image

(2) Extracting the candidates ofcorneal SR in search region

(3) Is there corneal SRin search region

(4) Defining regions of interest (ROIs)based on the detected corneal SRs

(6) Detecting pupil region in the ROIsby using subblock-based matching

(7) Selecting the area where thematching score is highest

as a pupil region

(8) Locating accurate pupil centerby ellipse fitting

(9) Selecting the region whose centeris closest to the center of pupil

as a corneal SR

(5) Detecting pupil region in thesearch region of corneal SR

by using subblock-based matching

No

Yes

Figure 3 Flowchart for detection of corneal SR and pupil regions

0 1 2

3 4 5

6 7 8

Figure 4 Example of subblock-based matching for pupil detection

To allow user-dependent calibration for eye tracking(Section 23) four NIR LEDs are attached at the four cornersof home appliances as shown in Figures 9(b)ndash9(d) Becausethe air conditioner switch is too small to be recognized bythe scene camera the NIR LEDs are attached at positionsslightly outside the four corners of the switch as shown inFigure 9(d)

Therefore the home appliances can be recognized onbasis of the patterns and the number of NIR LEDs detected

Subblock-based matchingusing 3 times 3 window mask (steps 5 and 6 of Figure 3)

Detected corneal SRfrom step 2 of Figure 3

ROI determinedon the basis of corneal SR

(step 4 of Figure 3)If the corneal SR is detected

in the search regionIf the corneal SR is not detected

in the search region

The search region ofleft corneal SR

for step 2 of Figure 3

The search region ofright corneal SR

for step 2 of Figure 3

Figure 5 Sample detection of the corneal SR and approximate pupilregions

by the NIR scene camera as shown in Figure 10 The spec-ifications of the NIR LEDs and NIR camera indicate thatonly the spots of the NIR-LEDs can be seen in the cameraimage Thereafter the NIR LEDs in the image captured bythe scene camera can be easily extracted by binarization

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

6 The Scientific World Journal

Centers of approximate pupil regionsobtained using subblock-based matching

ROIs defined on the basis of the centersof approximate pupil regions

Figure 6 Result of the sample pupil region detection

Binarization

Canny edge detection

Ellipse fitting

Detecting pupil center

Figure 7 Flowchart for the accurate detection of pupil centers

and component labeling Depending on their quantity andthe pattern of the spots the home appliances can then berecognized as shown in Figure 10 Different patterns can begeneratedwith theNIRLEDs thus different home appliancescan be recognized The proposed recognition method usingNIR LEDs and an NIR camera is robust to variations inexterior lighting conditions object size and viewpoints andhas a fast computation speed In Figure 10 the 119885-distance isthe distance between the userrsquos eye and the home appliance

23 Initial User-Dependent Calibration To calculate the gazeposition in the scene image an initial user-dependent cali-bration must be performed as shown in Figure 11 For thecalibration the user gazes at the four (upper left-hand upperright-hand lower right-hand and lower left-hand) corners ofthe home appliances at which the NIR LEDs are attached asshown in Figures 9(b)ndash9(d)

Using this calibration procedure the four center positionsof the pupil and corneal SR can be obtained as shown inFigure 11The corneal SR positions are used to compensate forhead movement when gazing at a point on a home appliance

In each image the position of the corneal SR is first setto the same position and the position of the pupil centeris moved by the amount of movement in the corneal SRNamely the position of the pupil center is compensated byensuring that the corneal SR position is the same in eachimage For example the positions of the pupil center andcorneal SR are (100 100) and (120 120) respectively in thefirst image In addition those of the pupil center and cornealSR are (130 90) and (110 130) respectively in the secondimage The deviations between the two corneal SR positionsare minus10 (110minus120) and +10 (130minus120) on the 119909- and 119910-axisrespectivelyTherefore if we try to set the corneal SR positionof the first image (120 120) to be the same as that of thesecond image (110 130) the pupil center position of the firstimage (100 100) is (90 110) on the basis of the disparities(minus10 +10)

If no corneal SR is detected as in the case of right eyesof Figures 5 and 6 the original pupil center positions areused for calibration and gaze estimation without compen-sation In this case the original pupil center represents theuncompensated position (by the corneal SR position) whichis detected using ellipse fitting as shown in Figure 7 In theabove example if the corneal SR is not detected in the firstimage the pupil center position of (100 100) is used forcalculating the gaze position

24 Calculation of Gaze Position on Scene Image Thefollowing four pupil center positions are obtained in theinitial calibration stage from Figure 11 (119875

1199090 1198751199100) (1198751199091 1198751199101)

(1198751199092 1198751199102) and (119875

1199093 1198751199103) (see left-hand side image of

Figure 12) The user actually gazes at the four corners of thehome appliance and these positions are observed in thescene camera image as (119874

1199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and

(1198741199093 1198741199103) (see right-hand side image of Figure 12) Hence

the relationship between the rectangles given by (1198751199090 1198751199100)

(1198751199091 1198751199101) (119875

1199092 1198751199102) and (119875

1199093 1198751199103) and (119874

1199090 1198741199100)

(1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103) can be given by a

geometric transform matrix as shown in (1) The final gazeposition can be calculated using this matrix and the detectedpupil and corneal SR centers as given by (2) [14 30 31]

As shown in Figure 12 the relationship between the tworectangles can be mapped using a geometric transform Theequations for this geometric transform are given as follows[14 30 31]

[[[[

[

1198741199090119874119909111987411990921198741199093

1198741199100119874119910111987411991021198741199103

0 0 0 0

0 0 0 0

]]]]

]

=

[[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]]

]

[[[[

[

1198751199090119875119909111987511990921198751199093

1198751199100119875119910111987511991021198751199103

11987511990901198751199100119875119909111987511991011198751199092119875119910211987511990931198751199103

1 1 1 1

]]]]

]

(1)

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 7

(a) (b)

(c) (d)

(e)

Figure 8 The procedure of accurately detecting pupil centers (a) Original image (b) Binarization image of (a) (c) Canny edge detectionimage of (b) (d) Ellipse fitting image of (c) (e) Image of detected pupil center

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

8 The Scientific World Journal

850nm

15 cm

15 cm

2 cm

(a)

785 cm

136 cm

(b)

34 cm

77 cm

(c)

12 cm

125 cm

(d)

Figure 9 (a) An NIR LED Attachment of NIR LEDs on home appliances such as (b) a 60-inch television (six NIR LEDs) (c) a heater(fiveNIR LEDs) and (d) an air conditioner switch (fourNIR LEDs) Note that the dotted circles indicate the NIR LEDs

The matrix coefficients 119886ndashℎ can be calculated from (1)Using these matrix coefficients one pupil position (1198751015840

119909 1198751015840

119910)

is mapped to a gaze position (119866119909 119866119910) on the scene image as

follows [14 30 31]

[[[

[

119866119909

119866119910

0

0

]]]

]

=

[[[

[

119886 119887 119888 119889

119890 119891 119892 ℎ

0 0 0 0

0 0 0 0

]]]

]

[[[[

[

1198751015840

119909

1198751015840

119910

1198751015840

1199091198751015840

119910

1

]]]]

]

(2)

In the proposed system the average gaze position of the leftand right eyes is considered to be the final gaze positionIf the final gaze position is within the region of the homeappliance as shown in Figure 13(a) for a predetermined timeperiod (2 s) the system determines that the user wants toselect the corresponding home appliance for control Thehome appliance area is the rectangular region defined by thefour spots of the NIR LEDs attached at the four corners Thedwell time (2 s) is experimentally determined considering theuserrsquos preference

3 Experimental Results

The performance of the proposed system was evaluatedwith two experiments The gaze estimation uncertainty was

analyzed in the first experiment and the accuracy with whichhome appliances are selected was determined in the secondexperiment A laptop computer (Intel Core i5 at 25 GHzwith 4GB of memory) was used for both experimentsThe proposed method was implemented using the OpenCVMicrosoft Foundation Class (MFC) library [32]

31 Gaze Estimation Uncertainty To calculate the gaze esti-mation uncertainty (the error of gaze estimation) in oursystem 10 users were asked to gaze at nine reference pointson a 60-inch television This procedure was repeated fivetimes The diameter of each reference point is 2 cm andthe 119885-distance between the usersrsquo eyes and the televisionwas approximately 300 cm Because only the gaze estimationuncertainty and not the accuracy of home appliance selectionis determined in this experiment only the television wasused as shown in Figure 14 In the initial user calibrationstage each user was asked to gaze at the fourNIR LEDsattached at the four corners of the television So the gazeestimation uncertainty was then measured when each usergazed at the nine reference positions (of Figure 14) whichare not used for user calibration and are not biased tothe calibration information Therefore the gaze estimationuncertainty can be measured more accurately Figure 14

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 9

(a) (b)

(c)

Figure 10 Results of recognition of home appliance in (a) Figure 9(b) at a 119885-distance of 300 cm (b) Figure 9(c) at a 119885-distance of 300 cmand (c) Figure 9(d) at a 119885-distance of 200 cm

shows the experimental setup used for measuring the gazeestimation error

The gaze estimation uncertainties (error) were calculatedfrom the difference between the calculated and the referencegaze positions as shown in Table 2 The average error isapproximately plusmn104∘

The gaze estimation uncertainty for the subjects rangesfrom 051∘ to 206∘ shown in Table 2 depending on theaccuracy of user calibration For instance user 8 correctlygazed at the four calibration positions (the fourNIR LEDsattached at the four corners of the television) in the initialuser calibration stage whereas users 4 and 5 did not

32 Accuracy of Home Appliance Selection To measure theaccuracy with which home appliances can be selected exper-iments were conducted for two cases users gazing and notgazing at a home appliance A total of 20 people participatedin the experiments Among them 18 were male and 2 werefemale and none of them wore glasses or contact lens Theaverage age (standard deviation) of the 20 participants is 274(185) and all of them are able-bodied

Generally good results cannot be easily obtained forpositions close to the borders of an appliance when comparedwith positions clearly within or outside the borders Inaddition the four positions outside each appliance cannotbe easily indicated Therefore the following schemes wereused in the experiments In the case when gazing at a homeappliance each user was instructed to randomly look at four

Table 2 Gaze estimation uncertainties in proposed system

User Error (standard deviation)User 1 plusmn091∘ (033)User 2 plusmn075∘ (008)User 3 plusmn068∘ (009)User 4 plusmn203∘ (037)User 5 plusmn206∘ (024)User 6 plusmn077∘ (016)User 7 plusmn096∘ (014)User 8 plusmn051∘ (014)User 9 plusmn092∘ (013)User 10 plusmn079∘ (012)Average plusmn104∘ (018)

positions within the borders of the home appliance withthe following instruction ldquojust look at any four positionsinside the home appliancerdquo In the case when not gazingat a home appliance the user was asked to randomly lookat four positions outside the borders of the appliance withthe following instruction ldquojust look at any four positionsclose to the home appliance (on the left- and right-handside and above and below the boundaries of the homeappliance)rdquo This procedure of looking at eight positions isone task and each participant repeated the task five times

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

10 The Scientific World Journal

(a) (b)

(c) (d)

Figure 11 Images of the detected centers of pupil and corneal SR when a user is gazing at the (a) upper left-hand (b) upper right-hand (c)lower left-hand and (d) lower right-hand corners of the 60-inch television

Upper-left Upper-right

Lower-left Lower-right

Scene imageEye image

(Px0 Py0) (Px1 Py1)

(Px3 Py3) (Px2 Py2)

(Ox0 Oy0)(Ox1 Oy1)

(Ox3 Oy3)

(Ox2 Oy2)

Figure 12 Mapping relationship between the pupil movable region (the rectangle given by (1198751199090 1198751199100) (1198751199091 1198751199101) (1198751199092 1198751199102) and (119875

1199093 1198751199103)) on

the eye image and the recognized region of the home appliance (the rectangle given by (1198741199090 1198741199100) (1198741199091 1198741199101) (1198741199092 1198741199102) and (119874

1199093 1198741199103)) on

the scene image

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 11

(a) (b)

Figure 13 Examples of (a) selection and (b) nonselection of a home appliance by gaze estimation Note that the red dotted circles indicatethe calculated gaze position

About 300 cm

Nine reference

About 55sim60 cm

points

Figure 14 Experimental setup for measuring gaze estimation error

Three home appliances were used in the experiments (60-inch television heater and air conditioner switch) fromthree viewpoints (left center and right) and two 119885-distances(television and heater 270 cm and 300 cm air conditionerswitch 170 cm and 200 cm) as shown in Figure 15 Theangles between the left and center (and the right and center)viewpoints are approximately 20∘ and 18∘ at 119885-distancesof 270 cm and 300 cm respectively Therefore 21600 eyeimages and 21600 scene images were obtained for theexperiments

Figure 16 shows the images of the different home appli-ances produced by the scene camera at different viewpointsand 119885-distances The upper and lower row images of (a)(b) and (c) of Figure 16 look similar because the differ-ence between the near and far 119885-distances (the upper and

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

12 The Scientific World Journal

Homeappliance

Figure 15 Top view of a sample experimental setup

lower row images of (a) (b) and (c) of Figure 16 resp) isminimal

The accuracy of selecting one of the three home appli-ances was calculated using only the pupil center (withoutcompensation with the corneal SR) and using both the pupilcenters and the corneal SR (with compensation) Henceforththese cases will be referred to as gaze estimation methods 1and 2 respectively

The accuracy of gaze estimation methods 1 and 2 wasquantitatively measured in terms of true positive rate (TPR)and true negative rate (TNR) True positive rate is the rateat which the calculated gaze position is located in the regioninside the home appliance boundary when the user is actuallygazing at the home appliance The TNR is the rate at whichthe calculated gaze position is in the region outside the homeappliance boundary when the user is not gazing at the homeappliance

For the 60-inch television the TPR and TNR values are6713 and 8813 respectively with gaze estimationmethod1 as shown in Figure 17 The average value of TPR and TNRis approximately 7763

For this appliance the TPR and TNR are 9929 and9938 respectively with gaze estimation method 2 asshown in Figure 18 The average value of TPR and TNR isapproximately 9934 By comparing with Figure 17 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR and TNR ofthe 20th user in Figure 18 are the lowest because of incorrectcorneal SR detection resulting from an elongated SR whichcauses the presence of SRs at the boundary of the iris and thesclera

For the heater the TPR and TNR values are 6238and 9238 respectively with gaze estimation method 1 asshown in Figure 19 The average value of TPR and TNR isapproximately 7738

In the experiment with the heater the obtained TPRand TNR values are 9975 and 9650 respectively with

gaze estimation method 2 as shown in Figure 20 Theiraverage value is approximately 9813 The accuracy of gazeestimation method 2 is clearly higher than that of gazeestimationmethod 1TheTPRandTNRof the 4th user are thelowest because the user did not correctly gaze at the cornersof the heater during the initial calibration

In the case of selecting the air conditioner switch thecalculated TPR and TNR values are 4233 and 9854 usinggaze estimationmethod 1 respectively as shown in Figure 21The average value of TPR and TNR is approximately 7044which is the lowest when compared with the other casesThisis because gaze estimationmethod 1 is significantly affected byheadmovements owing to the absence of compensation of thepupil center position by the corneal SR Because the switcharea is smaller than those of the TV and the heater evensmall headmovements significantly affect the gaze estimationposition

For this appliance the TPR and TNR values obtainedusing gaze estimation method 2 are 9892 and 9917respectively as shown in Figure 22 and their average isapproximately 9905 By comparing with Figure 21 we cansee that the accuracy of gaze estimation method 2 is higherthan that of gaze estimation method 1 The TPR of the 4thuser and TNR of the 13th user are the lowest because theseusers did not gaze at the correct corner positions of the switchduring the initial calibration By analyzing the images of pupilcenter and corneal SR position detection we can confirmthat all the detections are accurate Because no other factorcauses the low accuracy of TPR and TNR we estimate thatthe 4th and 13th users of Figure 22 did not gaze at the correctcorner positions of the switch during the initial calibrationalthough the positions which they actually looked at are notdifficult to be known In addition to analyze the effect ofinitial calibration on the accuracy the results of these subjects(the 4th and 13th users of Figure 22) were not excluded

Table 3 summarizes the results shown in Figures 17ndash22and the superiority of gaze estimationmethod 2 can be clearlyseen

33 Mode Transition of Our System In our system the signalfor moving the wheelchair can be triggered according to thedirection of gaze movement as shown in Figure 23(a) Ifthe gaze position is moved in the left- or right-hand sidedirection a signal is triggered for rotating the wheelchairin the left- or right-hand side direction respectively If thegaze position is moved in the upper or lower direction asignal is triggered for moving the wheelchair forward orbackward respectively If the gaze position is at the centralarea of scene camera image a signal is triggered for stoppingthe movement of the wheelchair As the future work ofadopting the electricalmotor on thewheelchair or interfacingwith electric wheelchair the user can reorient the wheelchairtoward the appliance of interest

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to move the direction of gaze position in thefive directions (left right upper and lower directions andcentral area resp as shown in Figure 24)This procedure was

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 13

(a)

(b)

(c)

Figure 16 Images of (a) 60-inch television (b) heater and (c) air conditioner switch produced by the scene camera with different viewpointsand119885-distances (note that the 1st and 2nd row images of (a)ndash(c) indicate the near and far119885-distances resp and the 1st 2nd and 3rd columnimages of (a)ndash(c) indicate the left center and right viewpoints resp)

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

14 The Scientific World Journal

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 3667 6750 8333 9667 9500 1417 8167 8083 7417 6917 9167 7750 6083 6917 7417 4167 4667 5250 7750 5167 6713TNR () 8083 8917 9250 9667 8833 8250 9167 9500 8833 9333 9833 9083 7917 8167 9750 8000 8250 7167 9000 9250 8813

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 17 Accuracy of selecting the 60-inch television using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 100 9917 100 100 100 100 100 100 100 100 100 100 100 100 100 100 8667 9929TNR () 100 100 100 100 9667 100 100 100 100 100 100 100 9500 100 100 100 100 100 100 9583 9938

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 18 Accuracy of selecting the 60-inch television using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 5750 6333 4917 8333 6750 6333 6667 4750 7583 3000 7000 8083 6417 5750 7417 4417 5000 6750 6917 6583 6238TNR () 9500 9500 9833 9083 9167 9417 9250 9833 9500 9833 9333 9750 8917 8167 9417 8250 8583 8833 8667 9917 9238

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 19 Accuracy of selecting the heater using gaze estimation method 1

iterated five times per each person Because it is difficult toindicate the reference position to be gazed in 3D space wetold only the instruction to each user for example ldquoif youwant to rotate the wheelchair in the left-hand side directionyou should move your gaze position in left direction Pleasemove your gaze position in the left directionrdquo Anotherexample is ldquoif youwant tomove thewheelchair backward you

should move your gaze position in lower direction Pleasemove your gaze position in the lower directionrdquo In order todiscriminate the userrsquos gaze direction we define the five areasin the image of frontal viewing camera as shown in Figure 24

Accuracy wasmeasured as correct recognition rate whichis calculated by the ratio of the number of correctly recog-nized trials to that of total trials For example if four trials

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 100 100 9583 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 9917 9975TNR () 9250 9667 9667 9000 100 9833 9833 9917 9583 100 9750 100 9083 9833 9750 9833 9500 9333 9167 100 9650

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 20 Accuracy of selecting the heater using gaze estimation method 2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 6667 3000 5750 4833 3417 3250 5833 1833 1417 4917 5417 3667 5917 2917 2167 4000 2083 5917 3917 7750 4233TNR () 100001000010000 9917 9750 9833 9500 1000010000100001000010000 9083 9917 9750 10000 9917 10000 9500 9917 9854

000100020003000400050006000700080009000

10000

Accu

racy

Users

Figure 21 Accuracy of selecting the air conditioner switch using gaze estimation method 1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 AvgTPR () 100 9500 100 8750 9917 100 9917 9750 100 100 100 100 100 100 100 100 100 100 100 100 9892TNR () 100 100 100 100 100 100 100 100 100 100 100 100 9167 100 9833 100 100 100 9583 9750 9917

7000

7500

8000

8500

9000

9500

10000

Accu

racy

Users

Figure 22 Accuracy of selecting the air conditioner switch using gaze estimation method 2

for moving the gaze position in the left direction are correctlyrecognized among total five trials the correct recognition rateis 80 (100 times 45 ()) Experimental results showed that foreach user and for each trial all the five gaze directions of thenavigationmodewere recognized correctly with the accuracyof 100

As shown in Figure 23(c) after the specific home appli-ance is selected by gazing at it for more than 2 s the control

panel with the 3 times 2menu options is displayed on the laptopcomputer placed below the gaze tracking system If the TV isselected the 3 times 2 menu options are power onoff selectionmode channel updown and volume updown as shownin Figure 23(c) If the selected home appliance is the heateror the air conditioner switch the 3 times 2 menu options arepower onoff selection mode air direction updown and airvolume updown The user gazes at one of the menu options

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

16 The Scientific World Journal

(a) (b)

(c)

Figure 23 Operation of the proposed system (a) Navigation mode for moving the wheelchair (b) Selection mode for determining the homeappliance to be controlled (c) Manipulation mode for controlling the selected home appliance

Table 3 TPR and TNR values (and standard deviation) of the three home appliances using gaze estimation methods 1 and 2 ()

Home appliance Accuracy Gaze estimation method 1 Gaze estimation method 2

60-inch televisionTPR (std) 6713 (021) 9929 (003)TNR (std) 8813 (007) 9938 (002)Avg (std) 7763 (014) 9934 (002)

HeaterTPR (std) 6238 (013) 9975 (001)TNR (std) 9238 (005) 9650 (003)Avg (std) 7738 (009) 9813 (002)

Switch for air conditionerTPR (std) 4233 (018) 9892 (003)TNR (std) 9854 (002) 9917 (002)Avg (std) 7044 (010) 9905 (003)

Avg (std) 7515 (011) 9884 (002)

for a specific time threshold which can be manually adjusteddepending on the user and the signal for controlling thehome appliance is triggered As the future work of generatingthe control signal of near-infrared (NIR) light using thecommunication board such as Arduino board [33] we canactually control the selected home appliance

In order to evaluate the reliability of our system theadditional experiments of five persons were performed Eachperson tried to gaze at six menus five times We told only theinstruction to each user for example ldquojust look at one of thesix menus one by onerdquo

Accuracy was measured as correct detection rate which iscalculated by the ratio of the number of correctly detectedtrials to that of total trials For example if three trials forgazing at the menu are correctly detected among total fivetrials the correct detection rate is 60 (100times35 ()) Exper-imental results are shown in Table 4 and we can confirmthat the reliability of our system in manipulation mode is

very high Because the number of menus of Figure 23(c) arelarger than that of regions of Figure 24 and there is largepossibility that the pupil is occluded by eyelid or eyelash bygazing at themenu in the downward direction of Figure 23(c)the accuracies of Table 4 are lower than those of navigationmode

Figure 25 shows the overall process of transitions betweenthe three modes implemented in our system navigation(moving the wheelchair in the direction of gaze movement)selection (selecting the specific appliance by gaze estimation)and manipulation (controlling the selected appliance bygazing at the control panel)

The transitions (from navigation to selection from selec-tion to navigation and from manipulation to navigation) aredone by eye closing within a specific time period The eyeclosure and openness is detected with the number of blackpixels within the detected eye region The transition (frommanipulation to selection) is done by gazing at the one menu

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 17

Table 4 Accuracies of trials of manipulation mode ()

User Menu positions AverageUpper-left Upper-center Upper-right Lower-left Lower-center Lower-right

User 1 60 80 100 100 100 100 90User 2 80 100 100 100 100 100 9667User 3 80 100 100 100 100 100 9667User 4 80 100 80 100 100 100 9333User 5 100 100 100 80 100 80 9333Average 80 96 96 96 100 96 94

Table 5 Accuracies of trials of transitions between modes ()

UserTransitions

AverageNavigation toselection

Selection tonavigation

Selection tomanipulation

Manipulation toselection

Manipulation tonavigation

User 1 100 80 100 100 100 96User 2 100 100 100 100 100 100User 3 100 80 100 100 80 92User 4 80 100 100 100 100 96User 5 100 100 100 80 80 92Average 96 92 100 96 92 952

Left Right

Upper

Lower

Center

Figure 24 Five areas in the image of frontal viewing camera fordiscriminating the userrsquos gaze direction in the navigation mode

of selection mode for a specific time threshold as shown inFigure 23(c)The transition (from selection to manipulation)is done by gazing at the specific home appliance for morethan the time threshold Because the kinds of the 3 times 2menu options of the control panel to be displayed can bedetermined after the specific home appliance is selected thereis no direct transition (fromnavigation tomanipulation) andit requires the intermediate transition to selection inevitably

In order to evaluate the reliability of our transition systemthe additional experiments of five persons were performedEach person tried to perform transitions (from navigationto selection from selection to navigation from selectionto manipulation from manipulation to selection and frommanipulation to navigation) five times by eye blink or gazingaccording to our instruction

Accuracy was measured as correct detection rate oftransition which is calculated by the ratio of the number ofcorrectly detected trials to that of total trials For exampleif four trials for performing transition are correctly detectedamong total five trials the correct detection rate is 80 (100times45 ()) Experimental results are shown in Table 5 and wecan confirm that the reliability of our transition system ofmodes is very high

In the initial calibration stage each user looks at thefour corners of a home appliance once and an additionalcalibration stage is not required A home appliance can beselected by looking at it for more than 2 s (this time thresholdcan be manually adjusted by the user) without additionalcalibration

34 Accuracy with People Imitating Involuntary Head Move-ments and Scalability of Our System It is difficult for usto have actual experiments with cerebral palsy patientswho suffer from involuntary head movements because ofthe difficulty involved in collecting the patients under theapproval of the institutional review board (IRB) Thereforethe experiments were conducted with two healthy personswho imitated involuntary head movements as shown inFigure 26 The diameter of each reference point is 6 cmExperimental results showed that the gaze estimation uncer-tainty (error) was plusmn24∘ with the standard deviation of 199Although this error was higher than that without involuntaryhead movements of Table 2 we can find that the navigationselection and manipulation process of Figure 25 can besuccessfully performed by our systemAndwe can expect thatour system can be used by people suffering from involuntaryhead movements

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

18 The Scientific World Journal

Navigation

Eye blinkGazing at the home appliancefor more than time threshold

Selection Manipulation

Eye blink Gazing at the menu of selection mode

for more than time threshold

Eye blink

Figure 25 Overall process of transitions between the three modes

The upper and lower row images of (a) (b) and (c) ofFigure 16 look similar because the difference between nearand far 119885-distances (the upper and lower row images of (a)(b) and (c) of Figure 16 resp) is not large In the case ofthe television and the heater the near and far 119885-distancesare 270 cm and 300 cm respectively In the case of the airconditioner switch they are 170 cm and 200 cm respectivelyAs shown in Figure 27 the size of the rectangle formed bythe four NIR LED spots at a far119885-distance of 350 cm is muchsmaller than that at near 119885-distance of 220 cm because thedifference between the near and far119885-distances is significant

As shown in Figure 16 different NIR LED patterns canbe produced using different quantities and positions of NIRLEDs Therefore many home appliances can be recognizedand consequently the scalability of our approach is highHowever when the distance between two NIR LED spotsis minimal because of the longer 119885-distance between thecamera and appliance (if the 119885-distance is greater than thecase of Figure 28 with the air conditioner switch at the 119885-distance of 375 cm) the NIR LED spots cannot be easilylocated thus the home appliance cannot be recognizedHowever the case where a user would want to select an airconditioner switch at a far 119885-distance considering typicaleyesight and typical home size is a rare event Therefore theproposed method is highly scalable and numerous homeappliances can be recognized using our method

4 Conclusion

In this study we proposed a novel interface system consistingof nonwearable eye tracking and scene cameras to select andcontrol home appliances The performance of the proposedsystem was investigated with three home appliances at differ-ent positions and 119885-distances In addition the performanceof two gaze-estimation methods was evaluated Because theproposed system enables users to control home appliances bylooking at them instead of using a display-based menu thisapproach might be more natural and convenient for users

In the future we would like to study the recognitionof multiple home appliances in images captured by a scenecamera Additionally wewill studymethods of enhancing the

Figure 26 Example of operation of the proposed system with theuser imitating involuntary head movement

Z 220 cm

Z 350 cm

Figure 27 Variation in rectangle size with 119885-distances

Figure 28 Frontal image of air conditioner switch at a 119885-distanceof 375 cm

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

The Scientific World Journal 19

accuracy of 3D gaze estimation by considering the 119911-axis inaddition to the 119909- and 119910-axes

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgments

This study was supported by the MSIP (Ministry of ScienceICT and Future Planning) Korea under the ITRC (Informa-tion Technology Research Center) Support Program (NIPA-2014-H0301-14-1021) supervised by the NIPA (National ITIndustry Promotion Agency) and in part by Basic ScienceResearch Program through the National Research Founda-tion of Korea (NRF) funded by the Ministry of Education(NRF-2012R1A1A2038666)

References

[1] C Mauri T Granollers J Lores and M Garcıa ldquoComputervision interaction for people with severe movement restric-tionsrdquo Human Technology vol 2 pp 38ndash54 2006

[2] J Tu H Tao and T Huang ldquoFace as mouse through visual facetrackingrdquo Computer Vision and Image Understanding vol 108no 1-2 pp 35ndash40 2007

[3] E Ko J S Ju E Y Kim and N S Goo ldquoAn intelligentwheelchair to enable mobility of severely disabled and elderpeoplerdquo in Proceedings of the International Conference onConsumer Electronics (ICCE rsquo09) Las Vegas Nev USA January2009

[4] C G Pinheiro Jr E L M Naves P Pino E Losson A OAndrade and G Bourhis ldquoAlternative communication systemsfor people with severe motor disabilities a surveyrdquo BioMedicalEngineering Online vol 10 article 31 2011

[5] B Rebsamen C L Teo Q Zeng et al ldquoControlling awheelchairindoors using thoughtrdquo IEEE Intelligent Systems vol 22 no 2pp 18ndash24 2007

[6] C Choi and J Kim ldquoA real-time EMG-based assistive computerinterface for the upper limb disabledrdquo in Proceedings of theIEEE 10th International Conference on Rehabilitation Robotics(ICORR rsquo07) pp 459ndash462 Noordwijkerhout Netherlands June2007

[7] L Y Deng C-L Hsu T-C Lin J-S Tuan and S-M ChangldquoEOG-based human-computer interface system developmentrdquoExpert Systems with Applications vol 37 no 4 pp 3337ndash33432010

[8] R Barea L Boquete S Ortega E Lopez and J M Rodrıguez-Ascariz ldquoEOG-based eye movements codification for humancomputer interactionrdquo Expert Systems with Applications vol 39no 3 pp 2677ndash2683 2012

[9] C-S Lin C-W Ho W-C Chen C-C Chiu and M-S YehldquoPoweredwheelchair controlled by eye-tracking systemrdquoOpticaApplicata vol 36 no 2-3 pp 401ndash412 2006

[10] T Kocejko A Bujnowski and J Wtorek ldquoEye mouse fordisabledrdquo in Proceedings of the Conference on Human SystemInteraction (HSI rsquo08) pp 199ndash202 Krakow Poland May 2008

[11] M-C Su K-C Wang and G-D Chen ldquoAn eye trackingsystem and its application in aids for people with severe

disabilitiesrdquo Biomedical Engineering Applications Basis andCommunications vol 18 no 6 pp 319ndash327 2006

[12] J J Magee M Betke J Gips M R Scott and B N WaberldquoA human-computer interface using symmetry between eyes todetect gaze directionrdquo IEEE Transactions on Systems Man andCybernetics B vol 38 no 6 pp 1248ndash1261 2008

[13] A Sesin M Adjouadi M Cabrerizo M Ayala and A BarretoldquoAdaptive eye-gaze tracking using neural-network-based userprofiles to assist people with motor disabilityrdquo Journal ofRehabilitation Research andDevelopment vol 45 no 6 pp 801ndash818 2008

[14] H C Lee W O Lee C W Cho et al ldquoRemote gaze trackingsystemon a large displayrdquo Sensors vol 13 pp 13439ndash13463 2013

[15] A Galante and P Menezes ldquoA gaze-based interaction systemfor people with cerebral palsyrdquo Procedia Technology vol 5 pp895ndash902 2012

[16] F Shi A Gale and K Purdy ldquoA new gaze-based interfacefor environmental controlrdquo in Universal Access in Human-Computer Interaction Ambient Interaction vol 4555 of LectureNotes in Computer Science pp 996ndash1005 2007

[17] S Nilsson T Gustafsson and P Carleberg ldquoHands free inter-action with virtual information in a real environment eyegaze as an interaction tool in an augmented reality systemrdquoPsychNology Journal vol 7 no 2 pp 175ndash196 2009

[18] D Mardanbegi and D W Hansen ldquoMobile gaze-based screeninteraction in 3D environmentsrdquo in Proceedings of the 1stConference on Novel Gaze-Controlled Applications (NGCArsquo11)Karlskrona Sweden May 2011

[19] J Turner A Bulling and H Gellersen ldquoExtending the visualfield of a head-mounted eye tracker for pervasive eye-basedinteractionrdquo in Proceedings of the 7th Eye Tracking Researchand Applications Symposium (ETRA rsquo12) pp 269ndash272 SantaBarbara Calif USA March 2012

[20] N Weibel A Fouse C Emmenegger S Kimmich and EHutchins ldquoLetrsquos look at the cockpit Exploring mobile eye-tracking for observational research on the flight deckrdquo inKarlskrona Sweden 7th Eye Tracking Research and ApplicationsSymposium (ETRA rsquo12) pp 107ndash114 Santa Barbara Calif USAMarch 2012

[21] J Hales D Rozado and D Mardanbegi ldquoInteracting withobjects in the environment by gaze and hand gesturesrdquo inProceedings of the 3rd International Workshop on PervasiveEye Tracking and Mobile Eye-Based Interaction pp 1ndash9 LundSweden 2013

[22] P Majaranta and K-J Raiha ldquoTwenty years of eye typingsystems and design issuesrdquo in Proceedings of the Symposium onEye Tracking Research amp Applications (ETRA rsquo02) pp 15ndash22New Orleans La USA March 2002

[23] C Lankford ldquoEffective eye-gaze input into windowsŮrdquo inProceedings of SymposiumonEyeTracking Research andApplica-tions pp 23ndash27 PalmBeachGarden Fla USANovember 2000

[24] R J K Jacob and K S Karn ldquoEye tracking in humanndashcomputer interaction and usability research ready to deliver thepromisesrdquo in The Mindrsquos Eye Cognitive and Applied Aspects ofEye Movement Research pp 573ndash605 Elsevier Oxford UK 1stedition 2003

[25] Webcam C600 2014 httpwwwlogitechcomen-ussupportwebcams5869

[26] C W Cho H C Lee S Y Gwon et al ldquoBinocular gazedetection method using a fuzzy algorithm based on qualitymeasurementsrdquo Optical Engineering vol 53 Article ID 0531112014

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

20 The Scientific World Journal

[27] B-S Kim H Lee andW-Y Kim ldquoRapid eye detection methodfor non-glasses type 3D display on portable devicesrdquo IEEETransactions on Consumer Electronics vol 56 no 4 pp 2498ndash2505 2010

[28] R C Gonzalez and R E Woods Digital Image Processing NewJersey NJ USA Prentice-Hall 2nd edition 2002

[29] L Ding andA Goshtasby ldquoOn the canny edge detectorrdquo PatternRecognition vol 34 no 3 pp 721ndash725 2001

[30] JW Lee H Heo and K R Park ldquoA novel gaze trackingmethodbased on the generation of virtual calibration pointsrdquo Sensorsvol 13 no 8 pp 10802ndash10822 2013

[31] S Y Gwon C W Cho H C Lee W O Lee and K R ParkldquoGaze tracking system for user wearing glassesrdquo Sensors vol 14pp 2110ndash2134 2014

[32] OpenCV httpopencvorg[33] Arduino 2014 httpenwikipediaorgwikiArduino

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of

International Journal of

AerospaceEngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation httpwwwhindawicom

Journal ofEngineeringVolume 2014

Submit your manuscripts athttpwwwhindawicom

VLSI Design

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation httpwwwhindawicom

Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

DistributedSensor Networks

International Journal of