Real-Time Computer Vision for Human Interfaces · EOG (electro-oculography) Figure: EOG...

EURON Summer School 2003

1

Real-Time Computer Vision for Human Interfaces

Yoshio MatsumotoNara Institute of Science and Technology, Japan


2

Introduction to Robotics Lab at NAIST

Nara


3

video


4

Face Measurement Systemand Its Applications

Yoshio MatsumotoNara Institute of Science and Technology, Japan


5

Motivation: Face Measurement

••

• Head motion and gaze direction reflect intention and attention of a person.

• For Human-Robot Interaction,

Goal of this research:

Head Motion => GestureGaze Direction => Attention

=>=>=> Intention

natural ways of communication is required.


6

• Head should not move, or head pose is measured by magnetic sensor etc.

• Eye rotation is detected using IR refection on corneal• Head mounted device prohibits natural behaviors • Accurate in best condition, however hard to keep it• Binocular systems and external camera system are also available

Gaze Tracking Techniques:Corneal Reflection


7

Gaze Tracking Techniques:Corneal Reflection (cont’d)

• Purkinje images appear as small white dots in close proximity to the (dark) pupil

• tracker calibration is achieved by measuring user gazing at properly positioned grid points (usually 5 or 9)

• tracker interpolates POR on perpendicular screen in front of user

Figure: Pupil and Purkinje images as seen by eye tracker’s camera


8

Gaze Tracking Techniques:EOG (electro-oculography)

Figure: EOG measurement:• relies on measurement of skin’s

potential differences using electrodes placed around the eye

• most widely used method some 20 years ago (still used today)

• measures eye movements relative to head position

• not generally suitable for POR measurement (unless head is also tracked)


9

Gaze Tracking Techniques: Scleral Contact Lens/Search Coil

Figure: Scleral coil:• search coil embedded in

contact lens and electromagnetic field frames

• possibly most precise• similar to electromagnetic

position/orientation trackers used in motion-capture


10

Gaze Tracking Techniques: Scleral Contact Lens/Search Coil (cont’d)

Figure: Example of scleral suction ring insertion:

• most intrusive method• insertion of lens requires care• wearing of lens causes

discomfort

• highly accurate, but limited measurement range (~5 )

• measures eye movements relative to head position

• not generally suitable for POR measurement (unless head is also tracked)


11

Real-Time Vision for Face Measurement

• One of the important topics in PUI research• Being actively studied at Microsoft, IBM, MIT,

CMU, Univ. of Illinois etc.

• Few of them can measure the quantitative gaze direction• Most of them use monocular camera systems


12

Our Approach

• Stereo camera system for 3D measurement• 3D facial model• Feature tracking by normalized correlation• 3D model fitting algorithm based on spring model

• Real-time Face Tracking• Gaze direction, blinking and lip motion are

additionally measured


13

1. Algorithm of Face Measurement


14

•••

System Configuration


15

3D Facial Model• Corners of eyes and mouth are used for tracking• 3D Facial Model = Template Images + 3D coordinates


16

3D Model Fitting for Face Tracking

)()(1

0ii

Tii

n

ii ytRxytRxWE −+−+=�

−

=

),,( zyxT),( ϕφθ ,R

ixiyiW

n

EMinimizing


17

d

3D Model Fitting for Face Tracking(cont’d)

),( ϕφθ ,R),( zyxT ,

)0,0,0(R)0,0,0(T

),( ϕϕφφθθ ∆+,∆+∆+R),( zzyyxxT ∆+,∆+∆+


18

d

3D Model Fitting for Face Tracking(cont’d)

),( ϕφθ ,−−−R

),( ϕφθ ∆,∆∆),( zyx ∆,∆∆

),( zyxT ,−−−


19

Result of Face Tracking


20

Accuracy of Face Tracking

Angle of Rotation Stage [deg]

Est

imat

ed A

ngle

[deg

]

0 20 40 60 80

0

20

40

60

80

Measurable Range

Pitch :-80 +80[deg]

Accuracy

Approx. 2[deg]

Yaw :-35 +35[deg]

Roll :-35 +35[deg]


21

Estimation of Gaze Direction


22

•

•

Result of Gaze Measurement


23



24



25

Result of Face Measurement


26

Hardware Configuration

Selectable from below combinations depending on requirements and costs

Desktop PC

Notebook PC

NTSC Camera 2NTSC Camera 2 (Multiplexed)

IEEE1394 Camera 2IEEE1394 High-Speed Camera (80Hz)


27

• Multiplexed stereo video streams into a single video stream.

• Vertical resolution of each video signal becomes half.

• Can be used for any conventional image processing system.

Field Multiplexing Device


28

The object that a person is looking at is recognized by calculating i for all objects.

Attention Recognition


29



30



31

Spotting gesture recognition based on Continuous DP Matching using head motions (velocity, angular velocity)

Gesture Recognition


32

Gesture Recognition


33

Gesture Recognition


34

Accuracy

Processing Speed

Specs of Developed System


35

Software Configuration


36

Software ConfigurationCommercial Face Recognition Software


37


Video Capturing Library


38


Application Software


39

Potential Application Areas

Since this system is non-contact, passive, andinexpensive, it can be applied to many application areas where conventional systems cannot be used.

• Human Interfaces• Computer Interface e.g. Hands-free mouse• Robot Interface e.g. Eye-contact communication• Safety System e.g. Driver support• Assistive Products for the disabled

• Human Modeling • Cognitive Science (Experiment on visual cognition• Ergonomics e.g. Human-friendly design)


40

2. Application to Human Interfaces


41

2.1 Computer Interfaces


42

Direct Usage of Head Movements:Hands-Free Mouse


43

How to Use Eye Movementsfor Computer Interfaces ?

• Eye movement [Glenstrup 95]– Convergence– Rolling– Saccades– Pursuit motion– Nystagmus– Drift and micro-saccades– Physiological nystagmus

• Need to refine raw data: – distinguish Fixations

from Saccades


44


Saccade/Fixation detection• Velocity-Threshold

– Saccades >300 deg/sec.– Fixations <100 deg/sec.– Usual threshold 200

deg/sec.

• Dispersion-Threshold

– Threshold set such that visual angle is between 0.5 ° and 1°.

( ) ( )( ) ( )yy

xxD

minmaxminmax

−−=


45

• Command based Interface– Obvious application: Selection of objects {Menu

selection, Window scrolling, …) � Pointing.– Midas Touch problem: Eyes not a control device.– Use dwell time to trigger a selection.

• Non-Command Interfaces– The computer monitors user’s actions instead of

waiting for user’s command.– Potential Applications: User Support

� Detect difficulties and provide translation support of difficult words



46

Pro-Active Dictionary:How to detect User Difficulties?

Gaze Pattern in

Normal Reading

Gaze Pattern when

Difficulties Encountered


47

Pro-Active Dictionary:Implementation

��

��

��

��

��

��

��

��

��

��

��


48

Pro-Active Dictionary:Demonstration


49

2.1 Robot Interfaces


50

Intelligent Wheelchair

PC

CCD cameras

Ultrasonic sensors

Laser Scanner

Notebook PC

CCD cameras


51

• Gesture– Nodding -> To start– Shaking ->To stop

• Face Direction– To determine the direction to move

• Gaze Direction– To determine if the user is concentrated


How is facial information used ?


52



53


Various lighting conditions


54



55


Sensor-based collision avoidance and wall following


56

Result of Experiment (Trajectory)

��

��

!"


57

Result (Input and Output Values)

��#��$��%��

#��&

��'�

( )

�"��*��+

!( !) ,( ,) -( -)

.(

/(

0(

,(

(

1,(

10(

1/(

1.(

*��+

0( 0) )(

2��

3�'�


58


-(

,(

!(

(

1!(

1,(

1-(

( -

�"��*��+

*��+

/ 4 !, !) !. ,! ,0

2��5�� &��

��6��7��6��7

How to detect concentration of the user?


59


Poster

Estimation of User’s Attention


60

Intelligent WheelchairEstimation of User’s Attention

Posters

Gaze direction

Robot


61

Paying attentionto the poster

Not paying attention to the poster

u Concentrated on the position of the poster.

u Distributed on various positions.



62



63

Receptionist Robot ASKA


64

Speech Recognitionand Voice Synthesis

Human andFace Recognition

NLP Dialog Model

Gesture Generation and

Autonomous Mobility

Receptionist Robot

Information Retrievalfrom Databases


Platform for experimentin real world


65

ASKA: System Overview

PC for gesture

Camera

Microphone

Speaker

PC for recognition and speech

PC for vision

PC for server


66

Hardware Configuration

telecamera

��1��"��

Base:tmsuk04 (tmsuk Co.,

LTD.)Degree of freedom

Chest 1 [DOF]Arms 14 [DOF] Hands 6 [DOF]


67



68



69

Problem to Solve with Vision

Ø

Ø

••


70


1. Human detection and tracking2. Detection of the sign of utterance


71

Pilot Study

••••••

•••

••••

Dialog Experiment

EU

RO

N Sum

mer School 2003

72

Information at U

tterance


73

Sign of Utterance : Start

The Face and gaze direct to object before the utterance


74

��

��

Sign of Utterance : Start

’•

•


75

Experiment : Detection of Utterance Sign


76

Demonstration : Interaction Based on Utterance Sign


77

3.Application to Human Modeling


78

Measurement of Infants


79



80


Measured Results

Time[sec]Face

Dir

ectio

n (H

oriz

onta

l)

Missing Data Measured Data


81

Measurement of TV Game Players


82

[s]

[s]

Frequency of Blinking

This result coincides with Psychological Report

Measurement of TV Game Players


83

System overview

Measurement of Drivers


84




85



86



87

Experiment at night



88

0

5

0

1

00100 50 0

Hor

izon

tal H

ead

Posi

tion[

mm

]

Time [min]

Yaw

Rate of V

ehicle [deg/sec]0 0.1 0.2 0.3

Ave. Speed 20[km/s] Ave. Speed 40[km/s]

Measurement of DriversHorizontal head movements at curves

Not passive head movements due to centrifugal force,but active head movements to cope with centrifugal force.

0 0.1 0.2 0.3


89

0 20 40 60 80

Subject A

Ver

tical

Hea

d Po

sitio

n [m

m]

Subject B Subject C

0 20 40 60 0 20 40 60

Vertical head position in long driving


0

50

100

Time [min] Time [min]Time [min]

•Body posture changed after long driving ?•Body lowered due to the softness of the seat?


90

Measured fixation point and yaw rate of vehicle



91




92




93

Where are you looking when lane changing?


Time[s]


94

Measurement of Patients in PET

To compensate head motion during measurement


95

SummaryHuman Interfaces

Robot InteractionComputer Interface

Face Measurement System

Human ModelingPsychology / ErgonomicsCognitive Science

Accuracy

Processing Speed


96

Fin

Real-Time Computer Vision for Human Interfaces · EOG (electro-oculography) Figure: EOG...

Documents

Transcript of Real-Time Computer Vision for Human Interfaces · EOG (electro-oculography) Figure: EOG...