Shape Recognitionfor Multi-Touch Table

Yehoraz Kasher Project Competition10/06/2012

Students: Itay Hubara

Amit Nishry

Supervisor: Yair Moshe

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion

Outline

2

Background

• Multi-touch is starting to appear in every day

computers

• New User Interfaces (NUI) are extensively

pursued

3

Motivation

• Existing multi-touch surfaces are expensive

• Current technology is unable to recognize

shapes

• Shape recognition enhances user interface

4

Goals

• Design a shape recognition algorithm

• Track and recognize combined shapes

• Integrate into open-source environment

• Real-Time performance

5

Challenges

• Noisy input image

• Real-time constrains

• Implementing into open-source environment

• Creating a complete system solution

• Shape combinations and movement

6

• Real-world view• IR input

Challenges (cont.)

7

Outline

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion8

System

• Hardwareo Multi-Touch Table

o FTIR Technology

• Softwareo Community Core Vision

o Client / Server Model

• Algorithmo Image Processing

o Shape Detection

o Tracking

• Applicationo Game

Hardware

9

Software

Algorithm

Application

FTIR - Frustrated Total Internal Reflection

Frustrated Total Internal Reflection

10

Multi-Touch Table

11

Outline

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion12

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification Tracking

LearningShape

ModelApplication

13

Learning Phase

On-line Phase

• Global thresholding leads to

inconsistent performance

• Each frame has a different

dynamic range

• Solution: Adaptive thresholding

Pre-Processing

Cropping

Stitching

Backgroundremoval

Noise filtering

Thresholding

Segmentation

14

Pre-processing (cont.)

IR input Background removed

15

Pre-processing (cont.)

Global threshold Adaptive threshold

16

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification

LearningShape

Model

17

Contour RegionStructural Global

Feature Extraction

Shape

features

[Zhang & Lu, 2004] 18

Shape

Region

Global

moments area axis …

• Area

• Major & minor axis

• Orientation

• Eccentricity

• 6 Hu moments

• 7 complex moments

Candidate Features

19

Image Moments

• Complex moments:

• Moments are centralized for translation

invariance

( , ) - image indicator function

,p q

pq

x y

c x iy x iy x y dxdy

20

• In polar coordinates:

• Rotation by

• In general

• Which leads to

2

1

0 0

,i p qp q

pqc r e r drd

Rotation Invariance

i p q

pq pqc e c

0

0n

j j jjk p q

1

j

j j

Nk

p q

j

I c

21

• Seven rotation and translation invariant

moments

• Useful for visual pattern recognition

• Rotation invariance via complex moments

1 11

2 20 02

c

c c

[Hu, 1962][Flusser, Suk & Zitová, 2009]

Hu Moments

22

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification

LearningShape

Model

23

Learning

Feature

selection

Shape

model

• Many features high computation

• Dimensionality reduction required

• Extract features at various locations

• Approx. 50 samples per shape

• Independent Gaussian model

24

Feature Selection• Exhaustive algorithm based on cross-validation

25

Features

First 10% of samples

Second 10% of samples

Tenth 10% of samples

Class

Third 10% of samples

Error 1

Error 2

Learning set

Test set

Test set

Learning set

Error 3

Error 10

Result: 1 2 10, , ,totale mean e e e

Feature Selection (cont.)

26

• Initialization o Set n=1 feature(s) out of a total of N features

• Stepo Calculate for all possible combinations

o Set n=n+1

• Stopping condition

o When

n

totaleN

n

1n n

total totale e

Feature Selection (cont.)

• Chosen features:o Area

o Major & minor axis

o Eccentricity

o 4 Hu moments:

• Empirical selection

• Similar to other shape recognition works

1 4 6 7, , ,

27[Hu, 1962][Sarfraz, 1993]

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification

LearningShape

Model

28

Classification

• Independent Gaussian distributions

• Naïve Bayes Classifier

•

29

2

1

1

1

1

( | ) ,

( | ,..., ) ( ( | )

ˆ classify( , , ) argmax ( | ,...,

)

)j

j i

j

m

j j

i i i

m i

m

j

j m

i

s

s f

s

s

p f N

p f f p p f

s f f

s

p

s

f fs

For each shape and feature

assume :

Naïve Bayes Classifier

• Advantages:o Generic

o Fast

o Allows thresholding

o Good performance

• Disadvantages:o Long offline learning process

30

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification Tracking

Application

31

Tracking

NN tracking

Combined

shapes

Internal

movement

False

positive

detection

• Nearest neighbors tracking

• Simplifies data handling

• Prevents false detections

• Allows complex situationso Shape combinations

o Internal movement in a combination

32

Complex Situations

• Shape combinations can appear as one

(legitimate) shape

• Possible classification mistakes:

33

Internal Shape Movement

• Complex shape modification

• Combined shape structure is maintained

34

Tracking

Segmentation

Classification

False Positive

Detection

Combined Shapes

Detection

Internal Movement Detection

Status Update

Blobs

Shapes

CombinedShapes

Blobs

Shapes

CombinedShapes

Current framePrevious frame

35

Video

36

Complex Situations

Shape Recognition

Pre-Processing

VideoFeature

ExtractionClassification Tracking

Application

37

Application

Client /

Server

Data

transfer

protocol

• Many possible applications

• Client / Server model established

• Community Core Vision

• Tangible User Interface Objects (TUIO) protocol for data transfer

38

Community Core Vision

• Open-source / cross-platform solution for

computer vision and machine sensing

• Supports multi-touch input

• Supports multi-camera feed

• Commonly used by developers

39

TUIO Protocol

• Open framework communication protocol and API for tangible multi-touch surfaces

• Supports various multi-touch informationo Location

o Orientation

o Acceleration

o …

• Commonly used with Community Core Vision

40

Outline

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion41

Implementation

42

• Simulations (Matlab)

• Implemented into CCV (C++)o Adaptive thresholding

o Learning process

o Classification

o Shape tracking

o Import / export classifiers

o Additions to GUI

• Implemented into TUIO (C++ & C#)o Sending shape information

o Distinguishing between touch and shape

Graphical User Interface

43

Learning

44

Outline

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion45

Tangram

• Chinese dissection puzzle with seven shapes

• The objective is to fill a given silhouette

• Demonstrates the algorithm capabilities

46

SIPL Tangram

47

Outline

Introduction

System

Algorithm

Implementation

Demo Game

Conclusion48

Demo

49

Performance Analysis

• Resourceso Intel i7 processor, 8 GB of memory

o Three IR cameras, 60 fps

• Real-Time performance

• Computing time < 2 miliseconds per frame

50

Summary

Multi-Touch table

Feature Extraction

Shape Classification

Tracking

Complex situations

CCV & TUIO

Tangram51

Shapes

Future Work

• Extending shape recognitiono Additional shapes

o Shape color

• Improving camera registration

• Cross-Platform implementations

• Finalize implementation of the demo game

52

Thanks

53

AviYairKobi

References

• D. Zhang and G. Lu, "Review of shape representation and

description techniques“, Pattern Recognition, vol. 37, pp. 1–19,

2004.

• M. K. Hu, "Visual Pattern Recognition by Moment Invariants“, IRE Trans. Information Theory, vol. 8, pp. 179–187, 1962.

• J. Flusser, T. Suk and B. Zitová, “Moments and Moment

Invariants”, John Wiley & Sons Ltd., 2009.

• C. M. Bishop, “Pattern Recognition and Machine Learning”,

Springer, 2007.

• "CCV, Community Core Vision“, [Online],

http://ccv.nuigroup.com/.

• "TUIO.org“, [Online], http://www.tuio.org/

54