Trinocular Stereo Vision Using a Multi Level Hierarchical ...
Lecture 25: Multi-view stereo, continued
description
Transcript of Lecture 25: Multi-view stereo, continued
![Page 1: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/1.jpg)
Lecture 25: Multi-view stereo, continued
CS6670: Computer VisionNoah Snavely
![Page 2: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/2.jpg)
Multi-view Stereo
Figures by Carlos Hernandez
Input: calibrated images from several viewpointsOutput: 3D object model
![Page 3: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/3.jpg)
The visibility problem
Inverse Visibilityknown images
Unknown SceneUnknown Scene
Which points are visible in which images?
Known SceneKnown Scene
Forward Visibilityknown scene
![Page 4: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/4.jpg)
Volumetric stereo
Scene VolumeScene Volume
VV
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Goal: Determine occupancy, “color” of points in VDetermine occupancy, “color” of points in V
![Page 5: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/5.jpg)
Discrete formulation: Voxel Coloring
Discretized Discretized
Scene VolumeScene Volume
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Assign RGBA values to voxels in Vphoto-consistent with images
![Page 6: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/6.jpg)
Complexity and computability
Discretized Discretized
Scene VolumeScene Volume
N voxelsN voxels
C colorsC colors
33
All Scenes (CN3)Photo-Consistent
Scenes
TrueScene
![Page 7: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/7.jpg)
Issues
Theoretical Questions• Identify class of all photo-consistent scenes
Practical Questions• How do we compute photo-consistent models?
![Page 8: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/8.jpg)
1. C=2 (shape from silhouettes)• Volume intersection [Baumgart 1974]
> For more info: Rapid octree construction from image sequences. R. Szeliski, CVGIP: Image Understanding, 58(1):23-32, July 1993. (this paper is apparently not available online) or
> W. Matusik, C. Buehler, R. Raskar, L. McMillan, and S. J. Gortler, Image-Based Visual Hulls, SIGGRAPH 2000 ( pdf 1.6 MB )
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel coloring solutions
![Page 9: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/9.jpg)
Reconstruction from Silhouettes (C = 2)
Binary ImagesBinary Images
Approach: • Backproject each silhouette
• Intersect backprojected volumes
![Page 10: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/10.jpg)
Volume intersection
Reconstruction Contains the True Scene• But is generally not the same
• In the limit (all views) get visual hull > Complement of all lines that don’t intersect S
![Page 11: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/11.jpg)
Voxel algorithm for volume intersection
Color voxel black if on silhouette in every image• for M images, N3 voxels
• Don’t have to search 2N3 possible scenes!
O( ? ),
![Page 12: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/12.jpg)
Properties of Volume Intersection
Pros• Easy to implement, fast
• Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000]
Cons• No concavities
• Reconstruction is not photo-consistent
• Requires identification of silhouettes
![Page 13: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/13.jpg)
Voxel Coloring Solutions
1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
> For more info: http://www.cs.washington.edu/homes/seitz/papers/ijcv99.pdf
3. General Case• Space carving [Kutulakos & Seitz 98]
![Page 14: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/14.jpg)
1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate
3.3. Color if consistentColor if consistent(standard deviation of pixel
colors below threshold)
Voxel Coloring Approach
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
![Page 15: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/15.jpg)
LayersLayers
Depth Ordering: visit occluders first!
SceneScene
TraversalTraversal
Condition: Condition: depth order is the depth order is the same for all input viewssame for all input views
![Page 16: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/16.jpg)
Panoramic Depth Ordering
• Cameras oriented in many different directions
• Planar depth ordering does not apply
![Page 17: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/17.jpg)
Panoramic Depth Ordering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 18: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/18.jpg)
Panoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 19: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/19.jpg)
Panoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 20: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/20.jpg)
Compatible Camera Configurations
Depth-Order Constraint• Scene outside convex hull of camera centers
Outward-Lookingcameras inside scene
Inward-Lookingcameras above scene
![Page 21: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/21.jpg)
Calibrated Image Acquisition
Calibrated Turntable360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
![Page 22: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/22.jpg)
Voxel Coloring Results
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
![Page 23: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/23.jpg)
Limitations of Depth Ordering
A view-independent depth order may not exist
p q
Need more powerful general-case algorithms• Unconstrained camera positions
• Unconstrained scene geometry/topology
![Page 24: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/24.jpg)
Voxel Coloring Solutions
1. C=2 (silhouettes)• Volume intersection [Baumgart 1974]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
> For more info: http://www.cs.washington.edu/homes/seitz/papers/kutu-ijcv00.pdf
![Page 25: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/25.jpg)
Space Carving Algorithm
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Repeat until convergence
• Choose a voxel on the current surface
• Carve if not photo-consistent
• Project to visible input images
![Page 26: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/26.jpg)
Which shape do you get?
The Photo Hull is the UNION of all photo-consistent scenes in V• It is a photo-consistent scene reconstruction
• Tightest possible bound on the true scene
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
![Page 27: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/27.jpg)
Space Carving Algorithm
Basic algorithm is unwieldy• Complex update procedure
Alternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardware
• Converges quickly in practice
• Easy to implement
Results Algorithm
![Page 28: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/28.jpg)
Space Carving Results: African Violet
Input Image (1 of 45) Reconstruction
ReconstructionReconstruction
![Page 29: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/29.jpg)
Space Carving Results: Hand
Input Image(1 of 100)
Views of Reconstruction
![Page 30: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/30.jpg)
Properties of Space Carving
Pros• Voxel coloring version is easy to implement, fast
• Photo-consistent results
• No smoothness prior
Cons• Bulging
• No smoothness prior
![Page 31: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/31.jpg)
ImprovementsUnconstrained camera viewpoints
– Space carving [Kutulakos & Seitz 98]
Evolving a surface– Level sets [Faugeras & Keriven 98]– More recent work by Pons et al.
Global optimization– Graph cut approaches
• [Kolmogoriv & Zabih, ECCV 2002]• [Vogiatzis et al., PAMI 2007]
Modeling shiny (and other reflective) surfaces– e.g., Zickler et al., Helmholtz Stereopsis
![Page 32: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/32.jpg)
Questions?
• 2-minute break
![Page 33: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/33.jpg)
Reconstructing Building Interiors from Images
Yasutaka Furukawa Brian Curless Steven M. SeitzUniversity of Washington, Seattle, USA
Richard SzeliskiMicrosoft Research, Redmond, USA
![Page 34: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/34.jpg)
Reconstruction & Visualizationof Architectural Scenes
• Manual (semi-automatic)– Google Earth & Virtual Earth– Façade [Debevec et al., 1996]– CityEngine [Müller et al., 2006, 2007]
• Automatic– Ground-level images [Cornelis et al., 2008, Pollefeys et al., 2008]
– Aerial images [Zebedin et al., 2008]
Google Earth Virtual Earth Zebedin et al.Müller et al.
![Page 35: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/35.jpg)
Reconstruction & Visualizationof Architectural Scenes
• Manual (semi-automatic)– Google Earth & Virtual Earth– Façade [Debevec et al., 1996]– CityEngine [Müller et al., 2006, 2007]
• Automatic– Ground-level images [Cornelis et al., 2008, Pollefeys et al., 2008]
– Aerial images [Zebedin et al., 2008]
Google Earth Virtual Earth Zebedin et al.Müller et al.
![Page 36: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/36.jpg)
Reconstruction & Visualizationof Architectural Scenes
• Manual (semi-automatic)– Google Earth & Virtual Earth– Façade [Debevec et al., 1996]– CityEngine [Müller et al., 2006, 2007]
• Automatic– Ground-level images [Cornelis et al., 2008, Pollefeys et al., 2008]
– Aerial images [Zebedin et al., 2008]
Google Earth Virtual Earth Zebedin et al.Müller et al.
![Page 37: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/37.jpg)
Reconstruction & Visualizationof Architectural Scenes
Google Earth Virtual Earth Zebedin et al.Müller et al.
Little attention paid to indoor scenes
![Page 38: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/38.jpg)
Our Goal• Fully automatic system for indoors/outdoors
– Reconstructs a simple 3D model from images– Provides real-time interactive visualization
![Page 39: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/39.jpg)
What are the challenges?
![Page 40: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/40.jpg)
Challenges - Reconstruction
• Multi-view stereo (MVS) typically produces a dense model
• We want the model to be– Simple for real-time interactive visualization of a
large scene (e.g., a whole house)– Accurate for high-quality image-based rendering
![Page 41: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/41.jpg)
Challenges - Reconstruction
• Multi-view stereo (MVS) typically produces a dense model
• We want the model to be– Simple for real-time interactive visualization of a
large scene (e.g., a whole house)– Accurate for high-quality image-based rendering
Simple mode is effective for compelling visualization
![Page 42: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/42.jpg)
Challenges – Indoor Reconstruction
Texture-poor surfaces
![Page 43: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/43.jpg)
Challenges – Indoor Reconstruction
Texture-poor surfaces Complicated visibility
![Page 44: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/44.jpg)
Challenges – Indoor Reconstruction
Texture-poor surfaces Complicated visibility
Prevalence of thin structures(doors, walls, tables)
![Page 45: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/45.jpg)
System pipeline
Images
Images
![Page 46: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/46.jpg)
System pipeline
Structure-from-Motion
Images
Bundler by Noah SnavelyStructure from Motion for unordered image collectionshttp://phototour.cs.washington.edu/bundler/
![Page 47: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/47.jpg)
System pipeline
Images SFM
![Page 48: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/48.jpg)
System pipeline
Images SFM
PMVS by Yasutaka Furukawa and Jean PoncePatch-based Multi-View Stereo Softwarehttp://grail.cs.washington.edu/software/pmvs/
Multi-view Stereo
![Page 49: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/49.jpg)
System pipeline
Images SFM MVS
![Page 50: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/50.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 51: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/51.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 52: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/52.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 53: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/53.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 54: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/54.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 55: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/55.jpg)
System pipeline
Images SFM MVS
Manhattan-world Stereo[Furukawa et al., CVPR 2009]
![Page 56: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/56.jpg)
System pipeline
Images SFM MVS MWS
![Page 57: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/57.jpg)
Manhattan-World Stereo (MWS)
0. Assume that surfaces are oriented along three mutually orthogonal directions
1. Axis-align model by detecting dominant orientations
2. Generate hypothesis planes for each direction
3. Label each pixel in each image with a plane (graph cuts)
![Page 58: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/58.jpg)
Finding hypothesis planes
![Page 59: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/59.jpg)
Example
Input image Computed depth map Mesh model
Texture-mapped model
![Page 60: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/60.jpg)
Manhattan-World Stereo (MWS)
Video
![Page 61: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/61.jpg)
System pipeline
Images SFM MVS MWS
Axis-aligned depth map merging(our contribution)
![Page 62: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/62.jpg)
System pipeline
Images SFM MVS MWS Merging
Rendering: simple view-dependent texture mapping
![Page 63: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/63.jpg)
Outline
• System pipeline (system contribution)• Algorithmic details (technical contribution) • Experimental results• Conclusion and future work
![Page 64: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/64.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 65: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/65.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 66: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/66.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 67: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/67.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 68: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/68.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 69: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/69.jpg)
Axis-aligned Depth-map Merging
• Basic framework is similar to volumetric MRF [Vogiatzis 2005, Sinha 2007, Zach 2007, Hernández 2007]
![Page 70: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/70.jpg)
Key Feature 1 - Penalty terms
Binary penalty
![Page 71: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/71.jpg)
Key Feature 1 - Penalty terms
![Page 72: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/72.jpg)
Key Feature 1 - Penalty terms
![Page 73: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/73.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
![Page 74: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/74.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)
![Page 75: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/75.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)
![Page 76: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/76.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)
![Page 77: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/77.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)• Smoothness (binary)
![Page 78: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/78.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)• Smoothness (binary)
![Page 79: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/79.jpg)
Axis-aligned Depth-map Merging
• Align voxel grid withthe dominant axes
• Data term (unary)• Smoothness (binary)• Graph-cuts
![Page 80: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/80.jpg)
Key Feature 2 – Regularization
• For large scenes, data info are not complete
![Page 81: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/81.jpg)
Key Feature 2 – Regularization
• For large scenes, data info are not complete
• Typical volumetric MRFs bias to general minimal surface [Boykov and Kolmogorov, 2003]
• We bias to piece-wise planar axis-aligned for architectural scenes
![Page 82: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/82.jpg)
Key Feature 2 – Regularization
![Page 83: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/83.jpg)
Key Feature 2 – Regularization
![Page 84: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/84.jpg)
Key Feature 2 – Regularization
![Page 85: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/85.jpg)
Key Feature 2 – Regularization
![Page 86: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/86.jpg)
Key Feature 2 – Regularization
![Page 87: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/87.jpg)
Key Feature 2 – Regularization
Same energy (ambiguous)
![Page 88: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/88.jpg)
Key Feature 2 – Regularization
Same energy (ambiguous)Data penalty: 0
![Page 89: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/89.jpg)
Key Feature 2 – Regularization
Same energy (ambiguous)Data penalty: 0 Smoothness penalty: Data penalty: 0 Smoothness penalty: 24Data penalty: 0
![Page 90: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/90.jpg)
Key Feature 2 – Regularization
shrinkage
![Page 91: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/91.jpg)
Key Feature 2 – Regularization
Axis-aligned neighborhood + Potts model
Ambiguous
Break ties with the minimum-volume solution
Piece-wise planar axis-aligned model
![Page 92: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/92.jpg)
Key Feature 3 – Sub-voxel accuracy
![Page 93: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/93.jpg)
Key Feature 3 – Sub-voxel accuracy
![Page 94: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/94.jpg)
Key Feature 3 – Sub-voxel accuracy
![Page 95: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/95.jpg)
Key Feature 3 – Sub-voxel accuracy
![Page 96: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/96.jpg)
Summary of Depth-map Merging
• For a simple and axis-aligned model– Explicit regularization in binary– Axis-aligned neighborhood system & minimum-
volume solution
• For an accurate model– Sub-voxel refinement
![Page 97: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/97.jpg)
Outline
• System pipeline (system contribution)• Algorithmic details (technical contribution)• Experimental results• Conclusion and future work
![Page 98: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/98.jpg)
Kitchen - 22 images1364 triangles
hall - 97 images3344 triangles
house - 148 images8196 triangles
gallery - 492 images8302 triangles
![Page 99: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/99.jpg)
Demo
![Page 100: Lecture 25: Multi-view stereo, continued](https://reader035.fdocuments.us/reader035/viewer/2022062314/568143f1550346895db07997/html5/thumbnails/100.jpg)
Questions?