Touchscreen Everywhere:

On Transferring a Normal Planar

Surface to a Touch-Sensitive Display

Jingwen Dai

Postdoctoral Research Associate

University of North Carolina at Chapel Hill

dai@cs.unc.edu

Motivation

04/11/2013

Bigger Display Portability

VS.

NTU IMI PhD Seminar 2/30

Motivation

04/11/2013 3/30

DLP Pico Projector

DC DV Mobile Phone

NTU IMI PhD Seminar

04/11/2013 NTU IMI PhD Seminar 4/30

~400mm Single Image 3D Action

Previews Works

Additional Sensors Light Touch (IR optical sensors)

Diamondtouch (capacitive sensor array)

Smartskin (mesh-shaped antenna)

Skinput (bio-acoustic sensing array)

LightSpace, Omnitouch (Kinect)

Computer Vision [Letessier2004] -- Fingertip tracking, not touching detection

[Kjeldsen2002, Hardenberg2001] -- Delay-based scheme

[Marshall2008] – Color change of the fingernail

[Song2007, PlayAnywhere2005] -- Shadow casted by finger

[Fitriani2007] -- Deformation on soft surface

04/11/2013 NTU IMI PhD Seminar 5/30

Overview

04/11/2013 NTU IMI PhD Seminar 6/30

Priors in Projector-Camera System

Geometric (Homography)

All these geometric priors can be derived through 2 projection-capture cycles in initialization stage.

04/11/2013 NTU IMI PhD Seminar 7/30

Embedding Codes into Video Projection

04/11/2013 8/30 NTU IMI PhD Seminar

R. Raskar, G. Welch, M. Cutts, A. Lake, L. Stesin, and H. Fuchs, “The office of the future”, SIGGRAPH98.

Embedded Pattern Design Strategy

Method Array Size Win. Size Alph. Length

[Morita 1988] 24 * 24 3 * 4 2

[Kiyasu 1995] 18 * 18 4 * 2 2

[Salvi 1998] 29 * 29 3 * 3 3

[Spoelder 2000] 65 * 63 2 * 3 2

[Albitar 2007] 27 * 29 3 * 3 3

[Desjardins 2007] 53 * 38 3 * 3 3

[Chen 2008] 82 * 82 3 * 3 7

04/11/2013 9/30

Summary of typical spatial coding methods

Constraints of Pattern Generation Code Uniqueness

Large Hamming Distance

NTU IMI PhD Seminar

Hand Segmentation

04/11/2013 10/30

A

A

B

C

C

NTU IMI PhD Seminar

Contrast Saliency

Region Discontinuity

Histogram-based Contrast Saliency

M. Cheng et al. Global contrast based salient region detection, CVPR2011.

04/11/2013 11/30 NTU IMI PhD Seminar

Mean-Shift Region Smoothing

04/11/2013 12/30 NTU IMI PhD Seminar

D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603–619, 2002.

Precise Segmentation by Fusing

04/11/2013 13/30

CF > Δ

NTU IMI PhD Seminar

Fingertip Detection

04/11/2013 NTU IMI PhD Seminar 14/30

Fingertip Candidates Detected Fingertips

Hand Contour Binary Hand Image

Touch Detection Through

Homography and Embedded Codes

04/11/2013 NTU IMI PhD Seminar 15/30

From Resistive Touch to Capacitive

Touch or Floating Touch

04/11/2013 16/30 NTU IMI PhD Seminar

Homology

Experiments – System Setup

04/11/2013 17/30 NTU IMI PhD Seminar

Experiments – System Initialization

Camera-Projector Homography Estimation

04/11/2013 18/30

Projected chessboard Captured image

NTU IMI PhD Seminar

Experiments – System Initialization

Camera-Table Homography Estimation

Four Correspondences: C1 ~ O, C2 ~ Px, C3 ~ Py, C4 ~ Pxy

04/11/2013 19/30

Credit Card Captured image

NTU IMI PhD Seminar

Experiments – Display Quality Evaluation

04/11/2013 20/30 NTU IMI PhD Seminar

Experiments – Hand Segmentation Accuracy

Evaluation

(a) Original Image

(b) Ground-truth

(c) Our Method

(d) SCM

(e) BkSub

(f) GB

04/11/2013 21/30 NTU IMI PhD Seminar

Experiments – Hand Segmentation Accuracy

Evaluation

04/11/2013 22/30 NTU IMI PhD Seminar

04/11/2013 NTU IMI PhD Seminar 23/30

Experiments – Touch Accuracy Evaluation

More HD videos http://cs.unc.edu/~dai/Research/TouchEverywhere/TouchEverywhere.htm

Experiments – Touch Accuracy Evaluation

04/11/2013

Comparison with recent depth-camera sensing based methods

In [Wilson 2010], the informal observed spatial error of finger detection on planar surface was between 3-6 pixels In [Omni-Touch 2011], the FRR and FAR of finger click detection on four different surfaces were 0.8% and 3.3%.

NTU IMI PhD Seminar 24/30

Experiments – Trajectory Tracking

Evaluation

04/11/2013 NTU IMI PhD Seminar 25/30

Experiments – Multi-Touch Evaluation

04/11/2013 NTU IMI PhD Seminar 26/30

Experiments – Efficiency Evaluation

04/11/2013 27/30

Subroutine Hand Seg.

FTip Loc.

Touch Det.

Total

Time (ms/frame) 14.63 1.32 1.74 17.69

NTU IMI PhD Seminar

Summary

Using only off-the-shelf devices

Precise hand segmentation in PROCAMS

Achieving 3D sensing without explicit 3D reconstruction

04/11/2013 NTU IMI PhD Seminar 28/30

04/11/2013 NTU IMI PhD Seminar

Related Publications Conference paper [1] J. Dai and R. Chung, Making Any Planar Surface into a Touch-sensitive Display by a Mere Projector and Camera, In Proc. of 9th IEEE International Workshop on Projector–Camera Systems (PROCAMS2012), pages 35-42, 2012.

[2] J. Dai and R. Chung, Combining Contrast Saliency and Region Discontinuity for Precise Hand Segmentation in Projector-Camera System, In Proc. of The 21st International Conference on Pattern Recognition (ICPR’12), pages 2161-2164, 2012.

Journal paper [3] J. Dai and R. Chung, Touchscreen Everywhere: On Transferring a Normal Planar Surface to a Touch-Sensitive Display, Submitted to IEEE Trans. on System, Man and Cybernetics (TSMC-B), November 2012. (under 2nd round review)

29/30

http://cs.unc.edu/~dai/Publications/Publications.htm

THANK YOU!!

04/11/2013 30/30 NTU IMI PhD Seminar