A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch InteractionQunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu

2013 10th IEEE International Conference on Control and Automation (ICCA)

2

Outline• Introduction

• Related Work

• Proposed Method• Hand localization

• Fingertips tracking

• The Multi-touch system

• Experimental Results

• Conclusion

3

Introduction

4

• Multi-touch technology:

• Sensor Based• Directly receive finger touch as input

• High cost → limits its application to some extent

• Computer Vision Based• Good scalability as well as good performance

Introduction

Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics and Applications, 2012

5

Introduction• In this paper:

• Propose a robust fingertip tracking algorithm:

• Real-time• Stereovision-based 3D multi-touch interaction system• Skin / Depth / Geometry structure

Hand Detection

FingertipTracking

Multi-touchSystem

6

Related Work

Related work• Geometry properties:

• Curvature• Edge or shape• Build a model

• Image Analysis• Template matching• Color Segmentation

7

L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007.

D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

Related work• Palm Center:

• Fingertip Detection

8

𝜴𝜺

[a]

[a]

[b]

[c] [d]

Geodesicdistance

GSP points

Neighbordepth

Related work• [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in

Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012

• [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011

• [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

• [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013

9

10

ProposedMethod

11

Hand Segmentation• Skin Color filter

• YCbCr color space

• Gaussian Mixture Model

• Describe the skin-color distribute• Single Gaussian Model:

• Gaussian Mixture Model:

Training data:

Weight of each Gaussian model:

color vector

12

Hand Segmentation• Skin Color filter

• : how skin-like the color is

• Expectation Maximization(EM) algorithm

13

Hand Segmentation• Depth Cue:

• The points with minimum depth are picked as seeds

• Region grow algorithm

skin

depth skin + depth

14

• Divide wrist and hand:• By a boundary curve [18]

• Minimum depth

• Boundary curve

Hand Segmentation

[18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2.

r : row indexc : column indexz(r,c) : depth value

,

range threshold(related to palm size)

boundary

15

Palm Region Extraction• Observation : Palm is a rectangle-like region

• Method : Project the hand region in all directions

𝟎° 𝟗𝟎° 𝐢𝐧𝐭𝐞𝐫𝐬𝐞𝐜𝐭𝐢𝐨𝐧𝐡𝐚𝐧𝐝

16

Palm Region Extraction

Intersection

17

Palm Center Localization•

𝟎°

>

>

𝟗𝟎°

X

𝟎°

>

𝟗𝟎°

、、

18

Palm Center Localization• Palm Center:

• The point with maximum distance from the closest palm boundary[18].

• The size of palm R:

palm region palm boundary

19

Fingertip Localization• Fingertip : The point with maximum distance to the palm center (on the contour of each finger)

• Candidate set F:P : contour pointC0 : palm centerd2 : distanceR : palm size

1.2

F

20

Fingertip Localization• Assign an index to each point in candidate set:

• Sort candidate set by

: index F : candidate setC0 : palm center the angle of with negative x-axis

21

Fingertip Localization• Distance between successive points :

• If > → Start & End point subset• Fingertips : maximum distance in each subset

22

Multi-touch system• TUIO (Tangible User Interface Object)

[24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and Simulation, 2005.

23

ExperimentalResults

24

Experimental Results• Xeon 3.07Ghz workstation

• frame rate : 20Hz on average (real-time)

• Modules• Fingertip tracking• TUIO server• TUIO client

[10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674

27

Conclusion

28

Conclusion• Fast and robust fingertip tracking• Without pressuring sensing device & extra marks

• Hand Segmentation• Depth / Skin

• Fingertip Detection• Palm region projection• Palm center distance from the boundary• Fingertip : assign index (angle)