Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections

Hui-Shan Kao

Advisor : Dr. Yi-Ping Hung

2

Outline

Introduction Related Work System Design Interaction Techniques Applications Conclusion

3

Outline

Introduction Related Work System Design Interaction Techniques Applications Conclusion

4

Introduction

Interaction with surfaces mainly support Multi-touch Tangible input

Interaction beyond surfacesCombine with mobile displays

iPad Tablet PCPico ProjectoriPhone

Introduction5

Two scenarios Type 1:

Using pico projector Multi-resolution presentation Augmenting personal information

Type 2: Using tablet PC

Interactive 3D viewer

Type2 Type1

Challenge6

How to know the 3D spatial relationship with surface

in real-time ??

Type2 Type1

7

Outline

Introduction Related Work System Design Interaction Techniques Applications Conclusion

8

Localization of Device

Enable 3D interaction on tabletop display Need to recognize the 6DOF of device

The way to know 6DOF Magnetic tracker

Penlight

H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009

9

Localization of Device

Vision based tracker Markers with known structure

Interactive handheld projector ARToolkit

Visible

Marker !

Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, 2007.D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: Proceedings of 12th Computer Vision Winter Workshop, 2007.

10

Outline

Introduction Related Work System Design

Hardware Configuration Interaction Techniques Applications Conclusion

11

 Hardware Configuration

Color Projection visible content

IR Camera IR

Camera

Color Projector

IR Camera

IR Projector

IR Projection invisible markers

Mirror

12

 Hardware Configuration

Multi-touch Surface Glass layer support touch force Diffuser layer display image

Placement of glass layer and diffuser layer Glass on top of diffuser Diffuser on top of glass

13

 Hardware Configuration

Option 1: Glass on top of diffuser

pico-projection reflection• degrade illumination• damage user’s eye

IR & colorprojectorIR camera

Pico-projector

IR camera

diffuser

touch-glass

Hard to Solve!

14

 Hardware Configuration

Option 2: Diffuser on top of Glass

IR-projection reflection• spot effect in IR images

IR & color projectorIR camera IR camera

Pico-projector

spot spot

touch-glass

diffuser

15

Hardware Configuration

Removing IR spot by using two cameras

Weighting Mask (2)IR Camera(2)

Weighting Mask (1)IR Camera(1)

Stitched View

IR Camera 1

IR Camera 2

16

Outline

Introduction Related Work System Design Interaction Techniques

Adaptive Markers for Camera Estimation Dynamic Markers for Multi-Touch

Applications Conclusion

17

3D Position Estimation

3D position?

Type2 Type13D position?

18

ARToolKitPlus for 3D estimation Fiducial marker

Self-identify by ID

Projecting a grid of markers on tabletop Each marker with

Unique ID Associated position

Compute the camera’s position based on the marker perceived

3D Position Estimation

(0,0)

(1,1)

ID = 0

19

Multi-Level Markers

Uni-level marker Camera might observe the markers too small or

too big

Multi-level marker System resizes the IR markers according to

camera position

20

Marker Split and Merge

Adapting the maker size Markers merge, when camera moves far

enough Markers split, when camera moves too

close Ensure camera to see at least 4 markers

21

Cooperating with multi-touch

Marker on for camera positioning Marker off for finger detection Foreground-ROI detection for marker

on/off

22

23

Cooperating with multi-touch

?

24

Cooperating with multi-touch

25

Background Simulation

Offline Save each marker as a patch image and

record the position of marker

_ = patch patch

collection

(offline) :: for each marker

base view

26

Background Simulation

Online As the layout re-arrange, the simulated

background can be built by the saving patch in real time

_ =simulate

d backgrou

nd

(online) :: for each marker in layout ( )

patch of marker

base view

27

Background Simulation

28

ROI Generation

1st frame after finger touch

2nd and the other frames

29

IR Projecto

r

IR Cameras

IR Camera of Mobile Device

Smoothing

Simulated Background

Observed Image

Foregrounds

Tangible Objects

Finger Touches

Layout Manager

Prediction

Color

Projector

ROI

Applications

KalmanFiltering

Real scene

Summary of Foreground Detection

30

Outline

Introduction Related Work System Design Interaction Techniques Applications Conclusion

31

Three Applications

Three applications provide intuitive and natural manipulation.

iLamp

iFlashlight

iView

32

iLamp

iLamp

Combine a pico projector and an IR camera

Project seamless high-resolution content, bringing more detailed information

Act as a desk lamp for personal use

iLamp34

? Real-timeRc Tc

Rpc Tpc

Rp Tp

[Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ]

Rpc Tpc

35

Pico projector and camera calibration

Off-line : Find transformation between camera and pico projector The projector can be viewed as a dual of camera Rpc Tpc are the Rotation and translation of the

attached camera

Rpc Tpc

36

On-line : Find transformation between pico projector and tabletop Estimate Rc Tc in real-time Compute Rp Tp

[Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ]

Pico projector and camera calibration

Rpc Tpc

37

iFlashlight

iFlashlight

A mobile version of iLamp, can be moved easily.

Multi-user cooperation

39

iView

An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives.

iView

41

Outline

Introduction Related Work System Design Interaction Techniques Applications Conclusion

42

Conclusion

A new interactive surface based on the programmable invisible markers.

Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera.

Bring another level of information on interactive surface.

43

Thank you