i 3D: Interactive Planar Reconstruction of Objects and Scenes
description
Transcript of i 3D: Interactive Planar Reconstruction of Objects and Scenes
![Page 1: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/1.jpg)
i3D: Interactive Planar Reconstruction of Objects and Scenes
Adarsh Kowdle Yao-Jen Chang Tsuhan Chen
School of Electrical and Computer EngineeringCornell University
09/25/2009WNYIP 2009
![Page 2: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/2.jpg)
2
i3D: Interactive planar reconstruction
![Page 3: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/3.jpg)
3
i3D: Interactive planar reconstruction
![Page 4: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/4.jpg)
4
Introduction to i3DAutomatic 3D reconstruction algorithms
Snavely et. al., SIGGRAPH ‘06
Furukawa et. al., CVPR ‘09
Sinha et. al., ICCV ‘09
Furukawa et. al., CVPR ‘07
![Page 5: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/5.jpg)
5
Introduction to i3DInteractive 3D reconstruction algorithms
Hengel et. al., VideoTrace
SIGGRAPH ‘07
Debevec et. al., SIGGRAPH ‘96
Sinha et. al., SIGASIA ‘08
![Page 6: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/6.jpg)
6
Introduction to i3DScribbles as a form of interaction
Srivatsava et. al., VMV ‘09
Li et al. Lazy Snapping. SIGGRAPH ’04.
Boykov and Jolly, ICCV ’01.
Improvement to Make3D (3D from a single image)
![Page 7: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/7.jpg)
Algorithm
![Page 8: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/8.jpg)
8
Preprocessing
Structure from motion
Superpixel map – mean shift
SIFT-like daisy descriptor
![Page 9: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/9.jpg)
9
Interactions
![Page 10: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/10.jpg)
10
Segmentation What do we do the scribbles?
One-against-all classifier – Logistic regression Feature vector – mean daisy descriptor of pixels in
each labeled superpixel Test each image – obtain response for each class Maintain smoothness in segmentation – energy
minimization problem
Node energy
Edge energy
![Page 11: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/11.jpg)
11
Segmentation
![Page 12: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/12.jpg)
12
Plane co-segmentation Transfer labels to 3D point cloud
Provide edge connectedness to help estimate a globally accurate plane
Estimate plane parameters – RANSAC
Estimate homography for each plane from scribbled image to other images
Warp these segments Also an energy minimization problem
![Page 13: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/13.jpg)
13
Plane co-segmentation
![Page 14: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/14.jpg)
14
Closing the loop
3D Model
![Page 15: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/15.jpg)
Visualizations
![Page 16: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/16.jpg)
16
Texture synthesis and rendering Texture preprocessing – segmentation
refinement #
View selection*
Incrementally build the model
Visualizations Dense point cloud Mesh visualization
*Debevec et. al., SIGGRAPH
’96
#Details in the paper
![Page 17: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/17.jpg)
17
Segmentation refinement
![Page 18: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/18.jpg)
18
Segmentation refinement
![Page 19: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/19.jpg)
19
Dense point cloud with texture Perform view selection and incrementally build the
model
Use camera projection matrix and back-project each pixel onto the corresponding plane
![Page 20: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/20.jpg)
20
Mesh visualization Modeling the 3D geometry using triangular
meshes Single mesh model – Single image texture
Multiple mesh model
![Page 21: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/21.jpg)
21
i3D: Interactive planar reconstruction More results…
![Page 22: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/22.jpg)
22
i3D: Interactive planar reconstruction More results…
![Page 23: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/23.jpg)
23
Conclusion i3D, a new interactive planar 3D
reconstruction algorithm using simple user interactions in the form of scribbles
Discuss multiple methods to refine segmentations and correct superpixel leaks
Complete and pleasing reconstructions – described two forms of visualizations
![Page 24: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/24.jpg)
Thank You
![Page 25: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/25.jpg)
![Page 26: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/26.jpg)
26
i3D: Interactive planar reconstruction More results…
![Page 27: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/27.jpg)
27
i3D: Interactive planar reconstruction More results…
![Page 28: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/28.jpg)
28
Closing the loop User provides scribbles on any image Perform segmentation of the image using the
daisy descriptors Provide edge connectedness scribbles Estimate plane parameters Perform plane co-segmentation Loop – Control with the user
![Page 29: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/29.jpg)
29
View selection Rank images based on angle between the
plane normal and viewing direction – lower the angle better the view
Incrementally build the model* Start with the best view for each plane – will have
holes due to occlusion Proceed with the next one in order of ranking and
incrementally fill up the holes
*Debevec et. al., SIGGRAPH ‘96
![Page 30: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/30.jpg)
30
Segmentation refinement Stage 1:
Warp segments from each image to every other image
For each image refine the segmentations of each plane by taking a per-pixel vote of all the warps
Stage 2: (mainly for scenes with polygon boundaries)
Use the line of intersection of every pair of planes
Project this line in 3D onto images Use line to define plane boundary
![Page 31: i 3D: Interactive Planar Reconstruction of Objects and Scenes](https://reader036.fdocuments.us/reader036/viewer/2022062301/56816150550346895dd0d641/html5/thumbnails/31.jpg)
31
3D Visualization Visualization chosen based on application
Full model required Dense point cloud with texture Multiple mesh model
Pleasing visualization – fly through Single mesh model