Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA....
Transcript of Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA....
![Page 1: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/1.jpg)
Reconstructing Building Interiors from Images
Yasutaka Furukawa Brian Curless Steven M. SeitzUniversity of Washington, Seattle, USA
Richard Szeliski
Microsoft Research, Redmond, USA
![Page 2: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/2.jpg)
Reconstruction & Visualizationof Architectural Scenes
• Manual (semi-automatic) approaches
– Google Earth & Virtual Earth
– Façade & CityEngine
Google Earth Virtual Earth City Engine
![Page 3: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/3.jpg)
Reconstruction & Visualizationof Architectural Scenes
• Manual (semi-automatic) approaches
– Google Earth & Virtual Earth
– Façade & CityEngine
• Automatic approaches w/ computer vision
Google Earth Virtual Earth AutomaticCity Engine
![Page 4: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/4.jpg)
Reconstruction & Visualizationof Architectural Scenes
What about indoor scenes?
![Page 5: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/5.jpg)
Reconstruction & Visualizationof Architectural Scenes
Relatively little attention given to indoor scenes
What about indoor scenes?
![Page 6: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/6.jpg)
What we do• Fully automatic system
– Starts from images
– Reconstructs a 3D model
– Provides real-time interactive visualization
![Page 7: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/7.jpg)
System pipeline
Images
Images
![Page 8: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/8.jpg)
System pipeline
Structure-from-Motion(Camera pose estimation)
Images
![Page 9: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/9.jpg)
System pipeline
Images
Structure-from-Motion(Camera pose estimation)
![Page 10: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/10.jpg)
System pipeline
Images Camera pose estimation
Multi-view Stereo(dense structure reconstruction)
![Page 11: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/11.jpg)
System pipeline
Images Camera pose estimation
Multi-view Stereo(dense structure reconstruction)
![Page 12: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/12.jpg)
System pipeline
Images Camera pose estimation Dense reconstruction
![Page 13: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/13.jpg)
System pipeline
Images Camera pose estimation Dense reconstruction
Mesh fitting
![Page 14: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/14.jpg)
System pipeline
Images Camera pose estimation Dense reconstruction Mesh fitting
![Page 15: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/15.jpg)
System pipeline
Images Camera pose estimation Dense reconstruction Mesh fitting
Image-based rendering
![Page 16: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/16.jpg)
Image-based rendering
View point
![Page 17: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/17.jpg)
Image-based rendering
View point
Reconstructedsurface model
![Page 18: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/18.jpg)
Basic Movement
Translation
Reconstructedsurface model
![Page 19: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/19.jpg)
Basic Movement
Translation
Reconstructedsurface model
![Page 20: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/20.jpg)
Basic Movement
Panning
Reconstructedsurface model
![Page 21: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/21.jpg)
Input image
Input image
Textureprojection
Textureprojection
Texture Mapping
Alpha-blending
![Page 22: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/22.jpg)
How it actually works
Input image
Input image
![Page 23: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/23.jpg)
How it actually works
Automaticsnapping
![Page 24: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/24.jpg)
![Page 25: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/25.jpg)
Demo
![Page 26: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/26.jpg)
Recap, Applications & Future work
• Fully automatic system
– From images
– To realistic visualization/virtual exploration
![Page 27: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/27.jpg)
Recap, Applications & Future work
• Fully automatic system
– From images
– To realistic visualization/virtual exploration
• Scaling up to
– A whole building with multiple floors
– Internet community photo collections
• Google streetview for indoor scenes
![Page 28: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/28.jpg)
Thank you - Any questions?
![Page 29: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/29.jpg)
Running Time
Kitchen (22 images) Hall (97 images) House (148 images) gallery (492 images)
SFM 13 76 92 716
MVS 38 158 147 130
MWS 39.6 281.3 843.6 5677.4
Merging 0.4 0.4 3.6 22.4
Running time of 4 steps [min]
![Page 30: Reconstructing Building Interiors from Images€¦ · Microsoft Research, Redmond, USA. Reconstruction & Visualization of Architectural Scenes •Manual (semi-automatic) approaches](https://reader033.fdocuments.us/reader033/viewer/2022060522/6050fbc7032ea20548554e4c/html5/thumbnails/30.jpg)
Acknowledgements
• Sameer Agarwal and Noah Snavely for support on SFM and discussion
• Funding sources– National Science Foundation grant IIS-811878
– SPAWAR
– The Office of Naval Research
– The University of Washington Animation Research Labs
• Datasets– Christian Laforte and Feeling Software for Kitchen
– Eric Carson and Henry Art Gallery for gallery