Discrete-Continuous Optimization for Large-scale Structure from Motion
description
Transcript of Discrete-Continuous Optimization for Large-scale Structure from Motion
![Page 1: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/1.jpg)
Discrete-Continuous Optimization for Large-scale Structure from Motion
David Crandall, Andrew Owens, Noah Snavely, Dan Huttenlocher
Presented by: Rahul Garg
![Page 2: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/2.jpg)
SfM from Internet Images• Recent work has built 3D models from large,
unstructured online image collections– [Snavely06], [Li08], [Agarwal09], [Frahm10], Microsoft’s PhotoSynth, …
• SfM is a key part of these reconstruction pipelines[Slide from authors]
![Page 3: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/3.jpg)
Camera 1
Camera 2
Camera 3R1,t1
R2,t2
R3,t3
p1
p4
p3
p2
p5
p6
p7
minimizef (R, T, P)
Structure from Motion
[Slide from authors]
![Page 4: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/4.jpg)
Feature detection
Bundle adjustmentRefine camera poses R, T
and scene structure P
Feature matchingFind scene points seen by
multiple cameras
InitializationRobustly estimate camera poses and/or scene points
Incremental Bundle Adjustment (IBA)
Reconstruction pipeline
[Slide from authors]
![Page 5: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/5.jpg)
How IBA Works• Image graph with pairwise point
correspondences
![Page 6: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/6.jpg)
How IBA Works
R1, t1 R2, t2
Start with a seed pair
Constraints fromcorrespondences
![Page 7: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/7.jpg)
How IBA works
R1, t1 R2, t2
Add new cameras to the seed
R3, t3
![Page 8: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/8.jpg)
How IBA works
R1, t1 R2, t2
Add new cameras to the seed
R3, t3
(x,y,z)
![Page 9: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/9.jpg)
How IBA works
R1, t1 R2, t2
Also calculate 3D positions of points that cameras see
R3, t3
(x,y,z)
![Page 10: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/10.jpg)
How IBA works
R1, t1 R2, t2
Wiggle solution periodically to get a better solution
R3, t3
(x,y,z)
![Page 11: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/11.jpg)
How IBA works
R1, t1 R2, t2
Wiggle solution periodically to get a better solution
R3, t3
(x,y,z)
![Page 12: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/12.jpg)
How IBA works
R1, t1 R2, t2
Wiggle (Bundle adjust) solution periodically to get a better solution
R3, t3
(x,y,z)
![Page 13: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/13.jpg)
How IBA works
R1, t1 R2, t2
Keep adding more cameras
R3, t3
(x,y,z)
R4, t4
![Page 14: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/14.jpg)
Problems with IBA
R1, t1 R2, t2
Incremental, prone to drifts and local minima
R3, t3
(x,y,z)
R4, t4
![Page 15: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/15.jpg)
Problems with IBA
R1, t1 R2, t2
Incremental, prone to drifts and local minima
R3, t3
(x,y,z)
R4, t4
![Page 16: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/16.jpg)
Problems with IBA
R1, t1 R2, t2
No easy way to incorporate priors (GPS info, etc.)
R3, t3
(x,y,z)
R4, t4
t4 from GPSR4 from Vanishing Points
![Page 17: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/17.jpg)
Initializing bundle adjustment
Incremental BAWorks very well for many scenes
Poor scalability, much use of bundle adjustmentPoor results if a bad seed image set is chosenDrift and bad local minima for some scenes
No way to add priors
[Slide from authors]
![Page 18: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/18.jpg)
Our Approach
R1, t1 R2, t2
Solve for everything simultaneously
R3, t3
(x,y,z)
R4, t4
![Page 19: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/19.jpg)
Our Approach
R1, t1 R2, t2
R3, t3
(x,y,z)
R4, t4
• Pose the problem as an MRF• Nodes – Cameras and 3D
Points• Labels – Discretized Pose and
3D locations
![Page 20: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/20.jpg)
The MRF model
binary constraints: pairwise camera transformations
unary constraints: pose estimates (e.g., GPS, heading info)
3D points can also be modeled
• Input: set of images with correspondence
[Slide from authors]
![Page 21: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/21.jpg)
Making MRF Tractable
Label Space:3 unknowns for rotation3 unknowns for translation
Dimensionality of State space: 6
![Page 22: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/22.jpg)
Making MRF Tractable
• First solve for rotation
Dimensionality of State space: 3 + 3
Label Space:3 unknowns for rotation3 unknowns for translation
![Page 23: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/23.jpg)
Making MRF Tractable
• First solve for rotation• Assume twist = 0
Dimensionality of State space: 2 + 3
Label Space:2 unknowns for rotation3 unknowns for translation
![Page 24: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/24.jpg)
Making MRF Tractable
• First solve for rotation• Assume twist = 0• Assume cameras are on the ground
Dimensionality of State space: 2 + 2
Label Space:2 unknowns for rotation2 unknowns for translation
![Page 25: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/25.jpg)
Making MRF Tractable
• First solve for rotation• Assume twist = 0• Assume cameras are near ground• Solve using parallelizable BP on a cluster, use distance
transforms to speed up message passing
Dimensionality of State space: 2 + 2
Label Space:2 unknowns for rotation2 unknowns for translation
![Page 26: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/26.jpg)
Refining the MRF Solution
• Use Non Linear Least Squares to refine the MRF solution (Discrete => Continuous space)
![Page 27: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/27.jpg)
Overall Approach
Discrete BP optimization over viewing directions
NLLS refinement of rotations
Discrete BP optimization over 2D translations
NLLS refinement of 3D translations
Pairwise reconstructions, geotags, VPs
Initial rotations
Refined rotations
Initial translations, 2D points
Initial 3D camera poses, 3D points
[Slide from authors]
![Page 28: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/28.jpg)
Feature point detection
Bundle adjustmentRefine camera poses R, T
and scene structure P
Image matchingFind scene points seen by
multiple cameras
InitializationRobustly estimate camera poses and/or scene points
SfM on unstructured photo collections
Discrete optimization over viewing directions
NLLS refinement of rotations
Discrete optimization over 2D translations
NLLS refinement of 3D translations
[Slide from authors]
![Page 29: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/29.jpg)
Experimental results• Evaluated on several Flickr datasets– Download photos (highest resolution available) &
geotags (if available) – Removed images with missing EXIF focal lengths– Removed panoramic and high-twist images
[Slide from authors]
![Page 30: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/30.jpg)
AcropolisReconstructed images: 454Edges in MRF: 65,097Median camera pose difference
wrt IBA: 0.1m
[Slide from authors]
![Page 31: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/31.jpg)
Dubrovnik (Croatia)Reconstructed images: 6,532Edges in MRF: 1,835,488Median camera pose difference wrt IBA: 1.0m
[Slide from authors]
![Page 32: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/32.jpg)
QuadTotal images: 6,514Reconstructed images: 5,233Edges in MRF: 995,734Ground truth for 348 camerasMedian error wrt ground truth:
1.16m (vs 1.01 for IBA)
[Slide from authors]
![Page 33: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/33.jpg)
Quad results
[Slide from authors]
![Page 34: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/34.jpg)
Central RomeReconstructed images: 14,754Edges in MRF: 2,258,416
[Slide from authors]
![Page 35: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/35.jpg)
Central RomeReconstructed images: 14,754Edges in MRF: 2,258,416Median camera pose difference
wrt IBA: 25.0m
Our result
IncrementalBundle Adjustment
[Agarwal09]
[Slide from authors]
![Page 36: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/36.jpg)
After discrete BP
[Slide from authors]
![Page 37: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/37.jpg)
After NNLS
[Slide from authors]
![Page 38: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/38.jpg)
After final BA
[Slide from authors]
![Page 39: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/39.jpg)
Running times• Our results are about as good as or better than
IBA, but at a fraction of the computational cost– Favorable asymptotic complexity– BP is easy to parallelize on a cluster, unlike BA
![Page 40: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/40.jpg)
IBA vs Our ApproachIBA
• Prone to drift, local minima
• Dependent upon seed set
• Not robust to outliers
• No easy way to add priors (GPS tags, etc.)
• Not Parallelizable
Our Approach
• Simultaneously solve for global optima
• Objective function robust to outliers
• Easy to add priors
• Parallelizable
![Page 41: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/41.jpg)
Discussion
Discrete BP optimization over viewing directions
NLLS refinement of rotations
Discrete BP optimization over 2D translations
NLLS refinement of 3D translations
Pairwise reconstructions, geotags, VPs
Initial rotations
Refined rotations
Initial translations, 2D points
Initial 3D camera poses, 3D points
Parallelizable! Easily incorporates noisy GPS info
Non Parallelizable
![Page 42: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/42.jpg)
Discussion
• Two step process for finding R and T. Is that why final BA is required?
• Final BA is still the bottleneck – not parallelizable
![Page 43: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/43.jpg)
Thank You
![Page 44: Discrete-Continuous Optimization for Large-scale Structure from Motion](https://reader036.fdocuments.us/reader036/viewer/2022062815/56816954550346895de0fd32/html5/thumbnails/44.jpg)
References• Photo Tourism: Exploring Image Collections in 3D. Noah Snavely, Steven M. Seitz, Richard Szeliski. SIGGRAPH 2006• Building Rome in a Day
Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski, ICCV 2009• Building Rome on a Cloudless Day
Jan-Michael Frahm, Pierre Georgel, David Gallup, Tim Johnson, Rahul Raguram, Changchang Wu, Yi-Hung Jen, Enrique Dunn, Brian Clipp, Svetlana Lazebnik, Marc Pollefeys, ECCV 2010