58 towards a new gaze tracker
-
Upload
aegis-accessible-projects -
Category
Technology
-
view
585 -
download
0
Transcript of 58 towards a new gaze tracker
VISION for ASSISTIVE TECHNOLOGIES
Edwige Pissaloux, ISIR/UPMC & CNRS (UMR 7222), Paris
Andrea Carbone, ISIR/UPMC & CNRS (UMR 7222), Paris
Christophe Veigl, FHTW, Vienna
Christophe Weiss, FHTW, Vienna
Institut des Systèmes Intelligents et de Robotique (ISIR)
U. Roma La Sapienza (IR) EC7T (Eye Com)Pertech (IR)RIT (IR)
Outline
1. Interaction and human capabilities : definition & new technologies.
2. Vision based technologies for interaction with pointing.3. AsTeRICS project contribution to vision technologydesign.4. Results of first user evaluations of AsTeRICS gaze tracker. 5. Conclusions.
UPMC
1. Interaction and humancapabilities :definitions & new technologies.
Definitions• Interaction = reciprocal actions/influences
- key concept of modern societies- based on human attention- involves different actors
(real person, virtual characters, objects, ubiquitous environments, etc.)
• Interaction allows- accessibility to all computer based ICT solutions- new skills acquisition & development- new communication modes elaboration
(ART= Attention Responsive Technology; multimodal paradigms, etc.).
New technologies for interaction
• New assistive technologies consider a mix of humancapabilities not integrated in classic interactive tools:- brain physiological signals (BCI)- brain plasticity- visual perception (visual attention)- pure technological « intelligent tools »
(tactile, haptic, robots, orthotics, computer vision).
This presentation objective : • visual perception & computer vision• pointing operation
2. Vision basedAT for interaction withpointing
• Pointing = the designation of an object by mediation of the arm, hand and sight.
• Two steps natural pointing operation usual execution : (1) pointed object localization in space and identification via sight
(point of regard (PoR) “touches” the object);
(2) arm/hand lifting for /physical or virtual (distal) “touch” of the object.
• In the case of upper limb impairments the second step should be executed by other means (other body parts).
New AT for pointingof an object on a PC
• Implementation outline
� use of image and vision processing
� targeted body part(s) detection and tracking
� mouse different operation simulation.
Finger as a pointer• Distal touch typing system (University of Tokyo)(1)
principle : fingertip spatial movements interpreted as different gestures
are linked to a usage of a standard mouse
(1) Niikura,T , Hirobe, Y. , Cassinelli, A. , Watanabe, Y. , Komuro,T. , Ishikawa, M. “In-air typing interface for mobile devices with vibration feedback”,
ACM International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2010).
Fingers are detected with a single high-frame-rate optical camera attached to a portable device of a
small (PDA-size) screen.
Mouse operations:
° mouse spatial displacements (by tracking the fingers)
° mouse click (by dwell time).
Limb as a pointer
• the concept of a pictorial structure (not fully connect graph) to represent the investigated part of the human body.
(1) Felzenszwalb, P.F., Huttenlocher, D.P. “Pictorial structure for object recognition”, Int. J. of Computer Vision, 61(1), pp. 55-79, 2005
(2) Siddiqui, M., Medioni, G., Robust Real-Time Upper Body Libm Detection and Tracking, VSSN’06, October 27, 2006, Santa Barbara
• The USC system (a monocular video stream)
- detects and tracks in the acquired images the human upper limbs (a tree structure)
- explores combinatorially using a body-global model the space of solutions over affine transforms and scales.
- optimizes the limb tracking via the gradient-descent approach
University of Southern California (USC)(2)MIT & Cornell (1)
Shoulder & elbowas a pointer
• EU 6th FWP IST IP AMI, University of Twente (1)- approach oriented at the estimation and recognition of poses
which generalizes detection of different body parts
(different limb sections such as shoulder, elbow)
Original image detected shoulders (and face) (2)
(1) Broekhuijsen, J., Poppe;, R., Peol, M. “Estimating 2D Upper Body Poses from Monocular Images”, Int. J. of Computer Vision, 54(1-3),
pp.181-207, 2006.
(2) Sidenbladh, H., Black, M. “Learning the statistics of people in images and video”, Int. J. of Computer Vision, 54(1-3):181-207, 2003.
• No published data on AT applications
Face and headas pointer
(1) Wahdi, M. , Koggalage, R. “Control of Mouse Movements using Human Facial Features”, 3rd Int. Conf. on Information and Automation
for Sustainability (ICAISP’2007), Melbourne, Australia, pp.13-18, 2007
(2) Gorodnichy, D., Roth, G., Nouse “use your nose as a mouse” perceptual vision technology for handfree games and interfaces,
Image and Vision Computing, 22(12), Oct. 2004, pp. 931-942.
• Different facial features : nose, global face, eyebrows, and their combination
• Processing- targeted feature is detected in acquired images during the calibration- targeted feature is tracked during the interaction.
Mouse functionalities emulation:- the mouse spatial displacements can be deduced from nose/head/eyebrows movements, - the mouse click (or object selection) is implemented through the (left or right)
eye double blinks or through dwell time.
(1) (2)
Eye& gaze as a pointer• Object selection via gaze is a fundamental interaction modality,
as the gaze position anticipates and finally allows, an action execution on the gazed object.
• Two configurations for eye- gaze trackers
Wearable (head mounted, Rochester Inst.) Remote (Tobii)
• The main characteristics :- remote systems : no devices/sensors have to be mounted on the subject’s body,
but restriction of the movement in the interaction space ; - Wearable systems : high accuracy, gaze-estimation in natural viewing context,
unlimited interaction space
3. AsTeRICSproject contribution to vision technology design (http://www.asterics.eu)
Objectives- the design of an adaptable and scalable system supporting
also unconventional peripherals (BCI, vision, etc.) for people with severely reduced motor capabilities interactions
- the evaluation of system with primary and secondary users.
AsTeRICS construction set (ACS) supports a wide range of user input and actuators (switches, grippers etc.)
that can be integrated with the vision-based system :
� tailoring the interaction scheme to the specific end-user needs and capabilities
� giving the possibility to explore completely new sensor combinations for human-computer interaction.
AsTeRICSwearable gaze trackerMain characteristics - adjustable to the end-user anatomy (head size, distance eye/camera, etc.) - adaptable to specific needs (small-amplitude head movements compensation,
easy to wear, precision of detection compatible with targeted skills for interaction, etc.)
Hardware - hot-mirrors or telescopic arms - multiple combinations of sensors coping with different capabilities and interaction
needs- batteries for long autonomy.
Foreseen combinations : - a three camera system using only visible lighting
for full gaze tracking in 3D space ;- a ‘minimal’ set with only one IR eye-camera and
a custom PCB integrating an IR tracker, a gyroscope and a Sip/Puff sensor.
AsTeRICS wearable gaze tracker (1)
Iris detection Visible & IR
Particle filter & Radial transform
Martinez, F., Carbone, A., Pissaloux, E., Radial Symmetry guided Particle Filter for Robust Iris Tracking, Proc. CAIP 2011, Spain
AsTeRICS gaze tracker
targeted applications � Wearable gaze tracker
• Vision « only » gaze tracker :
environmental control applications
• IR gaze tracker :
eye-movements interaction with a computer monitor
coupled with a range of actuators driven by the sip/puff sensor.
� Remote head/gesture tracker (webcam, Kinect)
• simple interaction purposes via head pose on low performance
computing platforms
• a nose- and chin-tracking for mouse simulation (mouse displacements
and an object selection/clicking) (Haar transform; KLT)
• Complex facial expression and gesture tracking (Active shape models)
ASM detection & tracking (left) ; landmark detection (right)
4. Results of first user evaluations.
AsTeRICS system prototype 1 – remote (web-camera based) gaze tracker- June-August 2011- different sensors and sensor-combinations - 50 users in Austria, Poland and Spain- spasms and involuntary head movements
Tests - Interaction with a screen
Results- spasms and involuntary head movements represented a big problem,
preventing precise pointing or computer mouse control- BUT highest level of acceptance.
Future developments- tremor reduction algorithms and - evaluation of the head mounted eye tracker
5. Conclusion
Vast impact of the vision technology a on the quality of life,
Vision allows new modes- to access ICT and internet-based services
(games, e-library, e-shopping, e-health, e-rehabilitation, e-learning, etc.)- to access smart environments which constitute the infrastructure for gaze interaction
with environmental control systems(lighting control, heating/ventilation or usage of home entertainment devices);
- for new skills acquisition (such as navigation in virtual worlds) (with training).
Future (second phase) of the AsTeRICS project- the head-mounted gaze estimation system will be finished, - both, the remote and the head-mounted solution,
will be evaluated in qualitative and quantitative user tests.
Thanks to- EU FP7 ICT program- France Soudage
Francis Martinez, ISIR/UPMCDarius Mazeika, ISIR/UPMC & Kaunas University (Lithuania)
Isabelle Liu, Master Student, UPMCJaza Gul Mohammed, Master Student, UPMC
Jacky Chen, E&M, MITFaith Keza, CS, MIT
Thankto youfor your participation