OverLay: Practical Mobile Augmented Reality Puneet Jain Duke University/UIUC Puneet Jain Duke...
-
Upload
charity-phelps -
Category
Documents
-
view
222 -
download
0
Transcript of OverLay: Practical Mobile Augmented Reality Puneet Jain Duke University/UIUC Puneet Jain Duke...
OverLay: Practical Mobile Augmented Reality
Puneet JainDuke University/UIUC
Justin ManweilerIBM Research
Romit Roy ChoudhuryUIUC
Idea
• Allow random indoor object tagging• Others should be able to retrieve
Faulty Monitor
Return CDs
Last year’s tax statements
Wish Mom Birthday
Mobile Augmented Reality
3
Introduction
WHY NOT A SOLVED PROBLEM?
Need to understand today’s approaches
Vision Sensing
Both necessary but not sufficient
-0.3
0.3
0.9
1.5
CLRLAY BRISK EDGEHIST MSER FAST CLRHIST
CLRGM ORB SIFT AKAZE SURF
La
ten
cy
(s)
Vision Sensing
Database Size0
0.2
0.4
0.6
0.8
1
10 20 40 60 80 100 120
La
ten
cy
(s
)
Accurate Algorithms are Slow
Feature Extraction
Feature Matching
6
Offloading + GPU
Network ≈ 302 ms
Extraction ≈ 29 ms
Matching ≈ 1 s
GPU on Cloud
For 100 image DB
Matching latency too high for real-time
Vision Sensing
7
Requires User Location
Requires Object Location
Brunelleschi's dome
Requires Precise Orientation
Not possible indoors
Vision Sensing
8
Accurate/Slow Quick/Inaccurate
Vision Sensing
Indoor LocationBut, Indoor Localization is not always available
Can accelerate Vision Prerequisites for Sensing
9
Location-free ARNatural pause, turn, walk indicate spatial-relationships between tags
Sensors can help in building such geometric layouts
Geometry, instead of location, can be used to reduce computation burden on vision
A
B
C D10 seconds
5 seconds
7 seconds
110°
80°
10
Primary Challenge: Matching Latency
Temporal Relationships Rotational Relationships
11
Temporal RelationshipsE D
C
B
A
T=0, saw A
T=21, saw E
T=7, saw B
T=15, saw CTemporal separations can be captured on cloud
TAB ≤ 7 + ETAB
TAB ≥ 7 – ETAB
TAC ≤ 15 + ETAC
TAC ≥ 15 – ETAC
ETAB, ET
AC, TAB, TAC≥ 0
TEMPORAL ROTATIONAL
12
Solving for Typical Time
TEMPORAL ROTATIONAL
EAB
Using temporal RelationshipsE D
C
ATCURRENT - TA
T=TA
T=TCURRENT
B
Time when the object is viewed
if ((TCURRENT – TA) + ETAB > TAB )
- Shortlist
TEMPORAL ROTATIONAL
14
Rotational RelationshipsE D
C
B
A
110° anti-clockwise
90° clockwiseGyroscope captures
angular changes
RB – RA ≤ 20° + ERBA
RB – RA ≥ 20° – ERBA
RC – RA ≤ 130° + ERCA
RC – RA ≥ 130° – ERCA
ERBA, ER
CA, RA, RB, RC ≥ 0
TEMPORAL ROTATIONAL
20° anti-clockwise
15
Using Rotational RelationshipsE D
B
A
RBRE
RD
RCURRENT = RA + Gyro
Gyro
RA
B’s rotational distance = RB – RCURRENT + ERB/2
- Pick tags closer in rotational distance
RB
RCURRENT
ERB/2
TEMPORAL ROTATIONAL
16
Annotation DBAnnotation DB
SURF(image, “Botanist”)
Retrieve
(frames, sensors)
Annotate
OverLay: Converged Architecture
Micro-trajectory
Spatial reasoning
Visual Geometry
Select Candidates
Selected candidates
GPU Optimized Pipeline
SURF RefineMatchframe
“Botanist”
Learning
Up
da
te m
od
ule
s
Macro-trajectory
Linear Program
Sensory Geometry
(time, orientation)
NETWORK
Blur?
Hand Motion?
Frame Diff?
This talk
17
Evaluation
Android App/Samsung Galaxy S4
11 Volunteers100+ Tags4200 Frame Uploads
Server: GPU on Cloud12 Cores, 16G RAM,6G NVidia GPU
18
System Variants
• Approximate (Quick Computer Vision)– Matching using approximate schemes e.g., KDTree
• Conservative (Slow Computer Vision)– Matching using brute-force schemes
• OverLay– Conservative + Optimizations
0 0.3 0.6 0.9 1.2 1.5 1.80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Approximate
Conservative
OverLay
Latency (seconds)
CD
F
Latency
Optimizations lead to 4 fold improvement
20
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Approximate
Conservative
OverLay
Precision
CD
FAccuracy: Precision
OverLay ≈ Bruteforce
21
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Approximate
Conservative
OverLay
Recall
CD
FAccuracy: Recall
Approximate < OverLay < Bruteforce
ConclusionVision and Sensing
based ARs
Geometric layouts: Accelerated Vision
OverLay: Practical Mobile AR
THANK YOU
synrg.csl.illinois.edu/projects/MobileAR
Puneet JainDuke University/UIUC
Justin ManweilerIBM Research
Romit Roy ChoudhuryUIUC
24
3D-OBJECTS
Handling 3D Objects: Learning
Tagged from particular angle Retrieving from different angle
26
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision BeforePrecision AfterRecall BeforeRecall After
Metric
CD
FAccuracy: After Learning
Recall > Bruteforce and Precision ≈ Bruteforce