Recent work in image-based rendering from unstructured image collections and remaining challenges...
-
Upload
vivien-jefferson -
Category
Documents
-
view
220 -
download
0
Transcript of Recent work in image-based rendering from unstructured image collections and remaining challenges...
![Page 1: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/1.jpg)
Recent work in image-based rendering from unstructured image collections and remaining challenges
Sudipta N. Sinha Microsoft Research, Redmond, USA
![Page 2: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/2.jpg)
• http://www.photosynth.net/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e
Image-based maps
![Page 3: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/3.jpg)
• Structure from motion (Sfm)
• Robust depth-map estimation
• Rendering
Key Steps
![Page 4: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/4.jpg)
• Structure from motion (Sfm)
• Robust depth-map estimation
• Image-based navigation
Recent results
A multi-stage linear approach to structure from motionSinha, Steedly & Szeliski, RMLE –ECCV workshop 2010
Piecewise planar stereo for image-based renderingSinha, Steedly & Szeliski, ICCV 2009
Image-based walkthroughs from incremental and partial scene reconstructions Kumar, Ahsan, Sinha & Jawahar, BMVC 2010
![Page 5: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/5.jpg)
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
![Page 6: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/6.jpg)
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pairPose estimation, triangulationRefinement
![Page 7: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/7.jpg)
Sequential SfmFitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02, Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pairPose estimation, triangulationRefinement
![Page 8: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/8.jpg)
Contributions• Vanishing point (VP) constraints reduces drift in rotations
– more accurate than [Govindu’04, Martinec’07] for urban scenes.– Faster pairwise matching + geometric verification
• New practical linear structure and translation estimation– more stable than the known linear method [Rother’03]– robust to outliers in 2D observations– easy to parallelize– faster than sequential Sfm
– much faster than L∞ - methods
Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE
workshop)
![Page 9: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/9.jpg)
Vanishing Point (VP) Detection
Pair Matching2 – VP + 2 point RANSAC VP tracks
relative rotationsFeature Extraction
VPs
interest pts
Images
GlobalRotation
Estimation
Linear Reconstruction
2-view Reconstruction
Robust Alignment
Global Scale & Translation
Estimation
VP tracks
relative rotations
global cameraorientations
relative pose estimates
Full Sfm initialization Final Bundle Adjustment
Linear multi-stage approach to structure from motion Sinha et. al. 2010 (ECCV-RMLE
workshop)
![Page 10: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/10.jpg)
Results
![Page 11: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/11.jpg)
Timings
![Page 12: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/12.jpg)
Break-up of Timings
![Page 13: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/13.jpg)
Comparison with sequential Sfm
![Page 14: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/14.jpg)
STREET sequence
HALLWAY sequence
OURS (65 cams, 52K pts)
before Bundle Adjustment BUNDLER (65 cams, 22K pts)
BUNDLER (139 cams, 13K pts) OURS (184 cams, 27K pts)
Comparison with sequential Sfm
![Page 15: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/15.jpg)
Piecewise Planar Stereo for image-based rendering
Graph-cut based energy minimization
Sinha et. al. ICCV 2009
![Page 16: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/16.jpg)
Piecewise Planar Stereo for image-based rendering
Sinha et. al. ICCV 2009
Planar Stereo Results
![Page 17: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/17.jpg)
also handle non-planar scenes now ...
Piecewise Planar Stereo for image-based rendering
![Page 18: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/18.jpg)
• Skip global scene reconstruction (Sfm) step, • Generate several overlapping, partial
reconstructions instead.• During navigation, jump
between local coordinate frames.• Scales easily, also parallelizable• Incremental matching & reconstruction
(images appear over time)
Image-based walkthroughs from incremental and partial scene reconstructions Kumar et. al. BMVC
2010
Fort sequence (~5800 images)
![Page 19: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/19.jpg)
• Accuracy vs. Connectedness• Reliable results from sparse, unstructured imagery
– wide-baseline matching is still difficult
• Representations: – metric vs. topological reconstructions ? hybrid ?
• Reconstructing Indoors– Bottlenecks: doorways, corridors.– fewer features, non-Lambertian surfaces
Existing issues in unstructured Sfm
![Page 20: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/20.jpg)
• Acquisition– Images vs. video– Short-term dynamics vs. long-term dynamics
• Need truly incremental Sfm– Start with scratch but keep going … ?– Interleaving matching, Sfm and dense stereo– Hybrid matching (2D—2D , 2D – 3D, 3D – 3D)
Dynamic Image-based Maps: Challenges
![Page 21: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/21.jpg)
• Temporal appearance changes– Illumination:
• day/night, seasons, weather, lights on/off• Cyclic, predictable
– Albedo changes• Store-fronts, ads-billboards,• irreversible
• Geometric changes: – temporary vs. permanent
• Mid-level features for higher level recognition
Dynamic Image-based Maps: Challenges
![Page 22: Recent work in image-based rendering from unstructured image collections and remaining challenges Sudipta N. Sinha Microsoft Research, Redmond, USA.](https://reader035.fdocuments.us/reader035/viewer/2022062423/56649ced5503460f949b96dc/html5/thumbnails/22.jpg)
Questions ?