Post on 23-Apr-2018
&
www.cea.fr
Aziz Dziri, Stephane Chevobbe, Mehdi Darouich
CEA LIST – Embedded Computing Laboratory – France
aziz.dziri@cea.fr
Gesture recognition on smart camera
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 2 &
Vision based gesture recognition
Targeted applications Smart buildings
Device control
Sign language recognition
Advantages Natural interaction
No additional expensive devices
Hitachi TV
Targeted systems Compact vision system
Low power < 10 W
Small size < 10 x 10 cm
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 3 &
Problematic
Constraints of embedded systems Input image : VGA grey level - QVGA RGB
Computing power : 50 – 700 MHz
Memory < 1 MByte
Existing gesture recognition algorithms Execution on PC platforms
Active sensor for some algorithms
Purpose : Study of the existing gesture recognition algorithms
Construction of gesture recognition pipelines
Evaluation of the pipelines on embedded processors
IcyCam by CSEM (44 mm2 – 80 mW)
Kinect camera
ToF camera
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 4 &
Overview
Context
State of art of gesture recognition algorithms
Implementation
Methodology and Results
Conclusions and future works
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 5 &
Overview
Context
State of art of gesture recognition algorithms
Implementation
Methodology and Results
Conclusions and future works
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 6 &
State of art
Hand gesture recognition pipeline Hand detection step
Gesture recognition step
Non-learning based methods
Learning based methods
Tracking step (not addressed)
Gesture recognition
Identified
gesture Features
extraction Classification
Hand detection
Input
image
Hand model
Segmentation localization
Learning
Hand
Region of Interest Gesture model
Learning
Hand tracking
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 7 &
State of art
Synthesis of the studied related work
Ref Detection method
Recognition method
(Classifier)
Recognition rate
(Number of gesture)
Image resolution
(Frame rate)
Size of feature vector
Computer
[1] Horimoto et al
2003
Background subtraction + skin color
threshold (Hue)
PCA (Nearest
neighbor)
83.7% (20)
160x120 (18 fps)
40 Celeron 1.3 GHz
[2] Zhao et al
2011
skin color thresholding (CbCr)+ Background
subtraction
HOG + PCA-LDA (Euclidean distance)
91% (10)
320x240 (15 fps)
9
2.67 GHz desktop
computer
[3] Liu et al
2009
Adaboost (learning machine)
invariant moments
(Multi-class SVM)
96.7% (7)
48x40 (-)
7
-
[4] Ankit et al
2009
Background subtraction
Finger tips detection
- (5)
- - 1.67GHz Intel Centrino Duo
processor
[5]Jae-Ho Shin et al 2006
Motion detection + entropy analysis
Hand edge in polar coordinate
95% (6)
320x240 (15 fps)
- Pentium PC
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 8 &
State of art
Complexity estimation 42x42 ROI size
HoG parameters [2]
Method ADD MUL DIV MOD ln sqrt arctan
With learning
HOG 11664 1296 5200 0 0 16 5184
Invariant Moment 49392 35321 8 0 6 1 0
Without learning
Finger tips detection 11 x m 12 x m m 2 x m 0 0 0
Hand edge in polar coordinate
3 x m + 5292
2 x m + 3528
m 0 0 m m
m = edge size
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 9 &
Gesture recognition pipelines
Learning based pipeline: Invariant Moment
Non-learning based pipeline: Finger tips detection
Color-based hand detection
Invariant Moment Recognition
Hand model
Gesture model
Finger Tips Detection
Gesture
Number of fingers
Color-based hand detection
Hand model
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 10 &
Overview
Context
State of art of gesture recognition algorithms
Implementation
Methodology and Results
Conclusions and future works
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 11 &
Detection
Color
conversion
Skin pixel
selection
Skin
histogram
Non-Skin
histogram
Histograms
normalization
Maximum
likelihood
Skin model
Segmentation Labeling ROI selection ROI
Skin pixels
Input image
Detection
Learning
Color-based hand detection
Invariant Moment Recognition
Filter
Off-line
On-line
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 12 &
Invariant Moments
Normalized
Central
Moments
Flusser
Invariant
Moments
Logarithm
transform ROI
gesture models
Manhattan
distance
Learning
Classification
Identified
gesture
Color-based hand detection
Invariant Moment Recognition
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 13 &
Fingertips
Chain code Peaks detection ROI Number of finger tips
K contour points K contour points
Color-based hand detection
Finger Tips detection
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 14 &
Fingertips
Chain code Peaks detection ROI Number of finger tips
K contour points K contour points
Color-based hand detection
Finger Tips detection
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 15 &
Overview
Context
State of art of gesture recognition algorithms
Implementation
Methodology and Results
Conclusions and future works
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 16 &
Methodology
Code transformations Floating point data coding not supported by some embedded processors
Floating point coding conversion to Fixed point coding
Maximum dynamic 64 bits
Operator transformation
Maximum resolution of Region of Interest
Invariant Moment : QQVGA (160x120)
Finger tips detection: QVGA (320x240)
Gesture recognition
Hand detection
Input
image
Identified
gesture
Gesture model
Hand model
Segmentation localization Features
extraction Classification
Learning
Hand
Region of Interest
Learning
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 17 &
Methodology
Methodology Quality vs execution time evaluation
C code
Camera
OpenCV
Visualization
Display
Intermediate
results
Processors
ARM - ANTX Tools
Execution
time
Fixed-point
PC Tools Quality
- Reference
- Degradation
Floating-point
ARM Cortex A9 : One core embedded processor without FPU ANTX : Low footprint control oriented processor without HW multiplier
OpenCV
acquisition
Hand
gesture
database
Fixed-point
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 18 &
Database
10 gestures - 40 occurrences
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 19 &
Results : Invariant Moments
Quality results : QQVGA (160x120) images ; 1 to 9 gestures
Execution performances 250 ms (4 fps) on ANTX @ 400 MHz
33ms (30 fps) on ARM Cortex A9 @ 400 MHz
Memory capacity < 150 kB
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 20 &
Results : Finger tips detection
Quality results QVGA images (320x240)
0 to 5 fingers to count
Reference (Floating point) : 80 % recognition rate
Embedded version (Fixed-point) : 79 % recognition rate
Execution performances 500ms (2 fps) on ANTX @ 400 MHz
100ms (10 fps) on ARM Cortex A9 @ 400 MHz
Memory capacity < 200 kB
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 21 &
Overview
Context
State of art of gesture recognition algorithms
Implementation
Methodology and Results
Conclusions and future works
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 22 &
Conclusions and future works
Two pipelines suitable for embedded context Learning based pipeline
Hand detection: Color based method
Gesture recognition: invariant moments
Non-learning based pipeline
Hand detection: Color based method
Gesture recognition: finger tips detection
Evaluation of both quality level and performances Invariant moments pipeline :
70% recognition rate for 4 gestures
4 to 30 fps
150 kB
Finger Tips pipeline :
79% recognition rate (0 to 5 fingers)
2 to 10 fps
200 kB
Quality : - Better classification step: Manhattan distance may be too simple - Use of peaks and valleys for finger tips detection
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 23 &
Conclusions and future works
Two pipelines suitable for embedded context Learning based pipeline
Hand detection: Color based method
Gesture recognition: invariant moments
Non-learning based pipeline
Hand detection: Color based method
Gesture recognition: finger tips detection
Evaluation of both quality level and performances Invariant moments pipeline :
70% recognition rate for 4 gestures
4 to 30 fps
150 kB
Finger Tips pipeline :
79% recognition rate (0 to 5 fingers)
2 to 10 fps
200 kB
Performances : - Optimization and parallelization of the code
-Tracking step
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 24 &
Next step …
Implementation of the pipelines on single chip vision system
IcyCam by CSEM (44 mm2 – 80 mW)
Color-based hand detection
Invariant Moment Recognition
Hand model
Gesture model
Finger Tips Detection
Gesture
Number of fingers
Color-based hand detection
Hand model
Centre de Grenoble 17 rue des Martyrs
38054 Grenoble Cedex
Centre de Saclay Nano-Innov PC 172
91191 Gif sur Yvette Cedex
Thank you Questions ?
Cliquez pour modifier le style du titre
WASC 2013| 03-04/June 2013 © CEA. All rights reserved | 26 &
References
[1] Horimoto, S., Arita, D., and Taniguchi, R., “Real-time hand shape recognition for human interface," in [Image Analysis and Processing, 2003.Proceedings. 12th International Conference on], 20-25 (sept. 2003).
[2] Zhao, Y., Wang, W., and Wang, Y., “A real-time hand gesture recognition method," in [Electronics, Communications and Control (ICECC), 2011 International Conference on], 2475-2478 (sept. 2011).
[3] Liu, Y. and Zhang, P., “Vision-based human-computer system using hand gestures," in [Computational Intelligence and Security, 2009. CIS '09. International Conference on], 2, 529-532 (dec. 2009).
[4] Ankit, G. and Kumar, A. P., “Finger tips detection and gesture recognition," tech. rep., Indian Institute of Technology, Kanpur (2009).
[5] Jae-Ho Shin, J.-S. L., “Hand region extraction and gesture recognition using entropy analysis," A 6, 216-222, IJCSNS (February 2006).