Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information...
Transcript of Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information...
![Page 1: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/1.jpg)
1
Vision-based 3D Media Synthesis and Display
Hideo SaitoProfessor
Dept. Information and Computer ScienceKeio University, Japan : pp. 1-31
French – Japanese Workshop in the field ofInformation and Communication Science and Technologies Nov.19, 2009
![Page 2: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/2.jpg)
Recent Trend for Visual Media Synthesis and Analysis
2
Multiple Camera
Virtualized Reality (CMU) Matrix
NFL EyeVision
Multiple Camera
Virtualized Reality (CMU) Matrix
NFL EyeVision
Virtual Reality
Mixed Reality Display
Virtual Reality
Mixed Reality Display
3D Display Technology
Free-Viewpoint VideoAuto-Stereoscopic DisplayVolumetric 3D Display
3D Display Technology
Free-Viewpoint VideoAuto-Stereoscopic DisplayVolumetric 3D Display
![Page 3: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/3.jpg)
3
• Reconstruction of 3D Structure from Multi-Views
• New View Synthesis from Recovered 3D Structure
– Free-Viewpoint Image Synthesis
– View Interpolation for autostereoscopic display
Multiple CameraMultiple Camera
![Page 4: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/4.jpg)
4
Input
sequence
Example:
3-Men Basketball
4D Model
(These are movies.)
Synthetic court
Virtualized Reality (CMU,1997-99)
![Page 5: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/5.jpg)
5
The 51-camera video
sequence are processed to
produce a complete 4-
dimensional (time + 3D)
description of an event.
A virtual video from
completely arbitrary view
points can be synthesized
from the 4D description,
including “placing” the
event in a “new”
environment, like a
synthetic gym.
3D Room
Virtualized Reality (CMU)
![Page 6: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/6.jpg)
6
Extension from Virtualized Reality
• Uncalibrated Cameras
• Real-Time
• More Cameras
• Extending Application
• Precise Shape Reconstruction
![Page 7: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/7.jpg)
7
For soccer scenes
cam 1
cam 2 cam 3
cam 4
cam1 cam2 cam3 cam4
[Inamoto and Saito, ICPR2002][Inamoto and Saito, IEEE Trans. MM, 07]
![Page 8: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/8.jpg)
8
GUI
Calculation of Calculation of Viewpoint PositionViewpoint Position
・・・
Virtual Views of The Stadium
Overlay on The Stadium Image
Arbitrary View Synthesis Arbitrary View Synthesis of Soccer Sceneof Soccer Scene
Rendering on Rendering on The StadiumThe Stadium ・
Multiple View Images Captured at A Stadium
Virtual View of The Dynamic Regions
Ref.Cam1 Ref.Cam2
![Page 9: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/9.jpg)
Real Camera 4
Real Camera 3
Real Camera 2
Real Camera 1 Cam 1-2 w = 2:8 Cam 1-2 w = 4:6 Cam 1-2 w = 6:4
Cam 1-2 w = 8:2 Cam 2-3 w = 2:8 Cam 2-3 w = 4:6
Cam 2-3 w = 6:4 Cam 2-3 w = 8:2 Cam 3-4 w = 2:8
Cam 3-4 w = 4:6 Cam 3-4 w = 6:4 Cam 3-4 w = 8:2
Example of Free Viewpoint Images
![Page 10: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/10.jpg)
Real Camera 1 Cam 1-2 w = 2:8 Cam 1-2 w = 4:6 Cam 1-2 w = 6:4
Real Camera 2Cam 1-2 w = 8:2 Cam 2-3 w = 2:8 Cam 2-3 w = 4:6
Real Camera 3Cam 2-3 w = 6:4 Cam 2-3 w = 8:2 Cam 3-4 w = 2:8
Real Camera 4Cam 3-4 w = 4:6 Cam 3-4 w = 6:4 Cam 3-4 w = 8:2
Example of Free Viewpoint Images
![Page 11: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/11.jpg)
11
Cam 1Cam 2
Cam 3 Cam 4
Example of Free Viewpoint Images
![Page 12: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/12.jpg)
12
Cam 1Cam 2
Cam 3 Cam 4
Example of Free Viewpoint Images
![Page 13: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/13.jpg)
13
Cam 1Cam 2
Cam 3
Example of Free Viewpoint Images
![Page 14: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/14.jpg)
14
Example of Free Viewpoint Images (HD)
![Page 15: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/15.jpg)
15
Research Issues
• Reduce the labor for camera calibration
• Real-time
• Accurate structure recovery
![Page 16: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/16.jpg)
16
Real-time depth imagevia plane sweeping [Nozick et.al 05]
(Univ. Paris-Est)
![Page 17: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/17.jpg)
17
Real time video-based rendering[Nozick, Saito 07] (Univ. Paris-Est)
16 Views are interpolated from 4 Cameras in Real time
![Page 18: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/18.jpg)
18
![Page 19: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/19.jpg)
19
[Hosokawa, Jarusirisawad, Saito, ICDSC09]
![Page 20: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/20.jpg)
20
[Jarusirisawad, Nozick, Saito, 3DIM09] (Univ. Paris Est)
![Page 21: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/21.jpg)
21
![Page 22: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/22.jpg)
22
Virtual Reality/Mixed Reality Display Virtual Reality/Mixed Reality Display
• Structure should be recovered in real-time
• Augmenting visual information should be synthesized in real-time
![Page 23: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/23.jpg)
23
overlay textures onto a deformable surface of an object in real time using a video see-through HMD.
Observer feels as if he was reading a real book.
E-Book display onto real paper surface [Emori, Saito ICAT03, IEEE-VR04]
![Page 24: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/24.jpg)
24
Result 1
Original image Overlay image
![Page 25: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/25.jpg)
25
A case that a book is turned upside down.
Overlay image 1 Overlay image 2
Result 2
![Page 26: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/26.jpg)
26
A case that we turn a page.
Page 1,2 Page 3,4
Result 3
![Page 27: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/27.jpg)
27
Original image Overlay image
A case that we see multiple books.
Result 4
These results(2~~~~4) shows that 2D matrix code is recognized correctly.
![Page 28: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/28.jpg)
28
Result
Overlay images generated by the system
![Page 29: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/29.jpg)
29
Projector
Camera
PC
Object
[Hirooka, Saito ISMAR03]
![Page 30: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/30.jpg)
30
Projection of the Image of Gutenberg Bible
![Page 31: Hideo Saito Professor Dept. Information and Computer ... · Hideo Saito Professor Dept. Information and Computer Science Keio University, Japan : ... cam1 cam2 cam3 cam4](https://reader030.fdocuments.us/reader030/viewer/2022020214/5b15d5d87f8b9a961e8b91c9/html5/thumbnails/31.jpg)
31
Tracking of position and pose for Mixed Reality
• Key Technologies
– tracking of 3D information from camera images