Geometry Directed Browser for Personal Photographs
description
Transcript of Geometry Directed Browser for Personal Photographs
![Page 1: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/1.jpg)
IIIT
Hyd
erab
ad
Geometry Directed Browser for Personal Photographs
Center for Visual Information Technology
IIIT Hyderabad
Aditya Deshpande, Siddharth Choudhary, P J Narayanan, Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar
![Page 2: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/2.jpg)
IIIT
Hyd
erab
ad
We use SfM and other 3D computer vision techniques to provide intuitive Geometry Directed Photo Browsing.
Photo-Browsing
• Digital Photography - No hard copy - Capture photographs and relive later on display device
• Photo-Browsers are tools to view digital photographs. E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc.
•Photo Browsing model has notevolved much.
![Page 3: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/3.jpg)
IIIT
Hyd
erab
ad
Related Work
• Face Detection & Tagging on Social Networking Sites.
• [Zhang et al. MM’03], Automatic annotation of family albums.
• [Davis et al. MM’05], Additional contextual data viz. time of capture, geo-tag, indoor/outdoor scene, co-occurring faces.
Above techniques only improve photo-browsing experience of social engagements.
![Page 4: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/4.jpg)
IIIT
Hyd
erab
ad
Our Goal
• Apart from social engagements, a large chunk of user’s personal photographs consist of tourist places & monuments.
• [Snavely et al. IJCV’08, SIGGRAPH’06] (Photosynth) - CPC Storage, local reconstruction to add new camera’s
• Choudhary et al., Li et al., Sattler et al., Irschara et al. etc. - Localize new query images w/o exhaustive search.
We combine SfM-Reconstruction + Localization to provide intuitive browsing of user photos in 3D space of the monument.
![Page 5: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/5.jpg)
IIIT
Hyd
erab
ad
Assumptions
• Our target platform is an off-the-shelf laptop or a desktop.
• User is expected to click around 5-50 photographs for a particular monument.
• The system should localize these user photographs in areasonable time.
• The system should provide a smooth visualization / transitions of all user photos and ~105 points of the monument.
![Page 6: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/6.jpg)
IIIT
Hyd
erab
ad
System Design(1) Heavy SfM Reconstruction done offline in the cloud
(2) GDBPackage : reconstruction + addnl. information downloaded to local disk
(3) User uploads personal photo’s through a camera / phone
(4) System registers user’sphotos to the point cloud andprovides 3D visualization.
![Page 7: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/7.jpg)
IIIT
Hyd
erab
ad
System Block Diagram
GDBPackage
User Photos
RegistrationModule
VisualizationModule
System is divided in two parts : 1. Registration / Localization Module2. Visualization Module
2 1
EstimatedCamera’s
![Page 8: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/8.jpg)
IIIT
Hyd
erab
ad
Localizing User Photo’s
• Trivial if photograph is taken from GPS enabled deviceand is geo-tagged!
• What if no geo-tag information?
• Two Localization Approaches :Image based search in a geo-tagged Image Dataset[Panda et al.] Geo-locate digital heritage site photos.
Using structure information in SfM Dataset[Irschara et al. CVPR’09], match to nearby similar images.[Li et al. ECCV’10], visibility prioritized 3D-2D matches. [Sattler et al. ICCV’11, ECCV’12], visual words to find 2D-3D matches.
![Page 9: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/9.jpg)
IIIT
Hyd
erab
ad
Localization - Choudhary et al.
• [Choudhary et al. ECCV’12]- Triangulate a seed point in the user photograph.- Further 3D-2D search is guided by visibility probabilities.- Find ~20 independent matches.- Use RANSAC to estimate camera parameters.
Probability Guided 3D-2D correspondence
3D Position
Up Vector
View Direction
![Page 10: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/10.jpg)
IIIT
Hyd
erab
ad
Advantages of Localization Method
• Data for Localization is stored in GDBPackage : (1) Cover Set (2) Visibility Matrix (3) Bi-Partite Visibility Graph
• CPC images need not be stored, data requirements are minimal.
• The method is fast and localizes images at the rate of 1sec/photo.
![Page 11: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/11.jpg)
IIIT
Hyd
erab
ad
Non-Localizable Photographs
• In some cases the images lack sufficient monument geometryfor localization to work :- Occluded by people.- Noisy images of nearby scenery/smaller monuments.- Zoomed in images of smaller monument structures etc.
Zoomed In View of Small Structure
(Pantheon Dataset)
Completely Occludedby People
(Colosseum Dataset)
![Page 12: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/12.jpg)
IIIT
Hyd
erab
ad
Non-Localizable Photographs
• Photographs have time of capture stored in their EXIF-tags.
• A non-localized image is placed at a position that is weightedaverage of its immediate known predecessor and immediateknown successor in time.
• Similarly, linear interpolation is also done for the view-direction vector to get the complete camera pose.
• The above method will not give the exact location, but placing it in temporal neighborhood suffices for display purposes.
![Page 13: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/13.jpg)
IIIT
Hyd
erab
ad
Visualization Module
• 3D Viewer
• Mouse Navigation
• Button Navigation
• Add Screenshot
• Delete Path
• Generate Photo-Tour
• 2D Viewer
![Page 14: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/14.jpg)
IIIT
Hyd
erab
ad
3D Photo Browser : Geometry Directed Photo-Browsing
• Initial Mode : 3D Model and small preview (thumbnails) of user photographs.
• Select Mode : Animate to clicked photo and detailed view.
• Linear quaternion interpolation of Rotation Matrix for smoothtransitions between images.
• Smooth transitions give a feel of the geometric space of the monument.
![Page 15: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/15.jpg)
IIIT
Hyd
erab
ad
3D Photo Browser : Generating Custom Photo Tours
• User can save the current viewpoint (“Add Screenshots”)
• Once a set of viewpoints are saved, he can smoothly animateover viewpoints. (“Generate Photo-Tour / Animate Path”)
• User can delete the viewpoints and generate a new photo-tour.
• Photo-Tours are a good way to creatively view personalphoto’s taken at a tourist place.
![Page 16: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/16.jpg)
IIIT
Hyd
erab
ad
ResultsMonument # Photos # Registered
PhotosReg. Time
(secs per photo)Colosseum 24 21 1.01Colosseum 19 14 0.97Pantheon 35 22 1.13
Stone Chariot(Hampi)
17 17 1.07
(a) Localization Module
(b) Visualization Module
![Page 17: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/17.jpg)
IIIT
Hyd
erab
ad
Conclusion and Future Work
• Minimal System Requirements.
• Intuitive 3D Visualization of User Photographs.
• Pipeline for 3D personal photo-viewing from SfM reconstruction.
• Port our system to a mobile phone and have a touch/gesture interface.
3D Photo-Viewing & Localization App
![Page 18: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/18.jpg)
IIIT
Hyd
erab
ad
Thank you. Questions?More Results
(a) Hampi Dataset (Stone Chariot)
(b) Pantheon Dataset
![Page 19: Geometry Directed Browser for Personal Photographs](https://reader036.fdocuments.us/reader036/viewer/2022062811/5681600f550346895dcf0fb1/html5/thumbnails/19.jpg)
IIIT
Hyd
erab
ad
Platform Details
Item Specification
CPU Intel ® CORE ™ i5
Clock Speed 2.44GHZ
RAM 4GB
GPU Intel ® HD Graphics Accelerator