OverLay: Practical Mobile Augmented Reality

Puneet JainDuke University/UIUC

Justin ManweilerIBM Research

Romit Roy ChoudhuryUIUC

Idea

• Allow random indoor object tagging• Others should be able to retrieve

Faulty Monitor

Return CDs

Last year’s tax statements

Wish Mom Birthday

Mobile Augmented Reality

3

Introduction

WHY NOT A SOLVED PROBLEM?

Need to understand today’s approaches

Vision Sensing

Both necessary but not sufficient

-0.3

0.3

0.9

1.5

CLRLAY BRISK EDGEHIST MSER FAST CLRHIST

CLRGM ORB SIFT AKAZE SURF

La

ten

cy

(s)

Vision Sensing

Database Size0

0.2

0.4

0.6

0.8

1

10 20 40 60 80 100 120

La

ten

cy

(s

)

Accurate Algorithms are Slow

Feature Extraction

Feature Matching

6

Offloading + GPU

Network ≈ 302 ms

Extraction ≈ 29 ms

Matching ≈ 1 s

GPU on Cloud

For 100 image DB

Matching latency too high for real-time

Vision Sensing

7

Requires User Location

Requires Object Location

Brunelleschi's dome

Requires Precise Orientation

Not possible indoors

Vision Sensing

8

Accurate/Slow Quick/Inaccurate

Vision Sensing

Indoor LocationBut, Indoor Localization is not always available

Can accelerate Vision Prerequisites for Sensing

9

Location-free ARNatural pause, turn, walk indicate spatial-relationships between tags

Sensors can help in building such geometric layouts

Geometry, instead of location, can be used to reduce computation burden on vision

A

B

C D10 seconds

5 seconds

7 seconds

110°

80°

10

Primary Challenge: Matching Latency

Temporal Relationships Rotational Relationships

11

Temporal RelationshipsE D

C

B

A

T=0, saw A

T=21, saw E

T=7, saw B

T=15, saw CTemporal separations can be captured on cloud

TAB ≤ 7 + ETAB

TAB ≥ 7 – ETAB

TAC ≤ 15 + ETAC

TAC ≥ 15 – ETAC

ETAB, ET

AC, TAB, TAC≥ 0

TEMPORAL ROTATIONAL

12

Solving for Typical Time

TEMPORAL ROTATIONAL

EAB

Using temporal RelationshipsE D

C

ATCURRENT - TA

T=TA

T=TCURRENT

B

Time when the object is viewed

if ((TCURRENT – TA) + ETAB > TAB )

- Shortlist

TEMPORAL ROTATIONAL

14

Rotational RelationshipsE D

C

B

A

110° anti-clockwise

90° clockwiseGyroscope captures

angular changes

RB – RA ≤ 20° + ERBA

RB – RA ≥ 20° – ERBA

RC – RA ≤ 130° + ERCA

RC – RA ≥ 130° – ERCA

ERBA, ER

CA, RA, RB, RC ≥ 0

TEMPORAL ROTATIONAL

20° anti-clockwise

15

Using Rotational RelationshipsE D

B

A

RBRE

RD

RCURRENT = RA + Gyro

Gyro

RA

B’s rotational distance = RB – RCURRENT + ERB/2

- Pick tags closer in rotational distance

RB

RCURRENT

ERB/2

TEMPORAL ROTATIONAL

16

Annotation DBAnnotation DB

SURF(image, “Botanist”)

Retrieve

(frames, sensors)

Annotate

OverLay: Converged Architecture

Micro-trajectory

Spatial reasoning

Visual Geometry

Select Candidates

Selected candidates

GPU Optimized Pipeline

SURF RefineMatchframe

“Botanist”

Learning

Up

da

te m

od

ule

s

Macro-trajectory

Linear Program

Sensory Geometry

(time, orientation)

NETWORK

Blur?

Hand Motion?

Frame Diff?

This talk

17

Evaluation

Android App/Samsung Galaxy S4

11 Volunteers100+ Tags4200 Frame Uploads

Server: GPU on Cloud12 Cores, 16G RAM,6G NVidia GPU

18

System Variants

• Approximate (Quick Computer Vision)– Matching using approximate schemes e.g., KDTree

• Conservative (Slow Computer Vision)– Matching using brute-force schemes

• OverLay– Conservative + Optimizations

0 0.3 0.6 0.9 1.2 1.5 1.80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Approximate

Conservative

OverLay

Latency (seconds)

CD

F

Latency

Optimizations lead to 4 fold improvement

20

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Approximate

Conservative

OverLay

Precision

CD

FAccuracy: Precision

OverLay ≈ Bruteforce

21

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Approximate

Conservative

OverLay

Recall

CD

FAccuracy: Recall

Approximate < OverLay < Bruteforce

ConclusionVision and Sensing

based ARs

Geometric layouts: Accelerated Vision

OverLay: Practical Mobile AR

THANK YOU

synrg.csl.illinois.edu/projects/MobileAR

Puneet JainDuke University/UIUC

Justin ManweilerIBM Research

Romit Roy ChoudhuryUIUC

24

3D-OBJECTS

Handling 3D Objects: Learning

Tagged from particular angle Retrieving from different angle

26

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Precision BeforePrecision AfterRecall BeforeRecall After

Metric

CD

FAccuracy: After Learning

Recall > Bruteforce and Precision ≈ Bruteforce