sensors in mobile devices for capturing panoramas and environment maps
description
Transcript of sensors in mobile devices for capturing panoramas and environment maps
SENSORS IN MOBILE DEVICES FOR CAPTURING PANORAMAS AND ENVIRONMENT MAPS
Maarten Van Lier2e Master Computerwetenschappen
Use Case Found an awesome view
View too large for one picture Take pictures with smartphone app
360° left to right / full spherical view App combines pictures into panorama
Reasonably fast
Problem Description Make panoramas and environment
maps Find alignment between image pairs Combine pictures
Problem Description (2) On smartphone!
Processing power = low Efficiency = necessary!
Interactivity Help user taking pictures
View result on smartphone Within a reasonable time
Standard approach Take pictures
Partially overlapping Find neighboring pictures Find alignment between neighbors
Intensity based alignment Feature based alignment
Composite pictures
Find AlignmentPixel Based Feature Based Find transformation
With lowest misregistration
With highest intensity match
Minimize error function Search options
Full search Hierarchical Incremental
Detect features Recognizable points Caracterized with vector SIFT, SURF, …
Match features Find same points in images Using distance between vectors
Find transformation Transforms features to
corresponing features RANSAC for outliers Find Homography
Smartphone Sensors But smartphones have sensors!
Accelerometer, compass, gyroscope Determine orientation of device
Use accelerometer & compass Use orientation!
For estimated picture location (and for real time “preview”)
The Overlap Approach Take pictures
Guide user with 3D preview of estimated panorama Save sensor data on shutter
Find overlapping regions Using saved sensor data
Detect & extract features From overlap regions
Match features Between corresponding overlaps
Find alignment Composite pictures
Taking Pictures
Taking Pictures
Taking Pictures
Taking Pictures
Find Overlap Regions Using estimated picture locations
To find neighboring pictures To determine estimated overlap
Bounding circle around picture center Bounding box around overlap region
Axis aligned vs non axis aligned
Find Overlap Regions (2)
Find Overlap Regions (3)
Find Overlap Regions (4)
Find Overlap Regions (5)
Find Overlap Regions (6)
Detect & Extract Features Only from overlapping regions of image Large overlap
Many features => good alignment Expensive detection & extraction
Small overlap Fewer features => bad alignment Cheaper detection & extraction
For actual panoramas & env maps Expected: large overlap regions Actual speed gain may not be very large
Match Features Neighboring pictures
Using estimated locations (sensor data) Features in same overlap region Features estimated to be close to
eachother Use estimated 3D or polar feature locations But not yet implemented
Match Features
Likely match
Unlikely match
Likely match
Possible false match
Overlap Regions Test
Overlap Regions Test (2)
Overlap Region Test (3) Recommended for most panorama apps:
About 20% on each side=> about 70-80% when on all sides
But only an estimate=> needs to be investigated & tested!
Here: on average 62% overlap Because overlap at (nearly) all sides
Expected: a 30-40% drop in time when extracting features only from overlap regions
Results: Timings (6 pictures)Standard Approach Overlap Approach
Find features 2383 ms for 4998
features Matching features &
calculate homographies 18661 ms
Total: 21044 ms
Calculate Overlap Regions 9 ms for 12 overlaps
Find Features 1617 ms for 3153
features Matching features &
calculate homographies 8252 ms
Total: 9878 => 53% less!
Results Standard Approach
Results Overlap Approach
So… What’s Next? Improve stitched result
Increase overlap region Paper! Second semester:
Optimizations Port to smartphone
Check timings on smartphone More testing & results Try something similar for pixel based alignment
Initial location using sensors + incremental refinement And maybe HDR Writing thesis text, making poster
References Image Alingment and Stitching: A
Tutorial (Richard Szeliski, 2006)
SURF: Speeded Up Robust Features (Herbert Bay, Tine Tuytelaars, Luc Van Gool,
2006) Recognising Panoramas
(M. Brown, D. G. Lowe, 2003) BoofCV
http://boofcv.org
Questions and suggestions?