1 Establishing the Independent System & Market Operator (ISMO)
IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D...
Transcript of IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D...
![Page 1: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/1.jpg)
IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction
Soshi Shimada 1,2, Vladislav Golyanik 2,3, Christian Theobalt 3 , and Didier Stricker 1,2
1. Augmented Vision, DFKI 2. University of Kaiserslautern 3. MPI for Informatics
![Page 2: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/2.jpg)
2
Motivation
![Page 3: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/3.jpg)
3
Motivation- 3D reconstruction of a deformable object from monocular 2D image sequences is
still a challenging problem
![Page 4: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/4.jpg)
4
Motivation
2D RGB image 3D point clouds
- 3D reconstruction of a deformable object from monocular 2D image sequences is still a challenging problem
![Page 5: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/5.jpg)
5
Related works
![Page 6: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/6.jpg)
6Figure1. NRSfM technique (Golyanik et al. , 2017)
• Non Rigid Structure from Motion (NRSfM)
Related works
![Page 7: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/7.jpg)
7Figure1. NRSfM technique (Golyanik et al. , 2017)
• Non Rigid Structure from Motion (NRSfM)- input: point tracks on 2D frames
Related works
![Page 8: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/8.jpg)
8Figure1. NRSfM technique (Golyanik et al. , 2017)
• Non Rigid Structure from Motion (NRSfM)- input: point tracks on 2D frames- basically no limitation regarding target objects
Related works
![Page 9: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/9.jpg)
9Figure1. NRSfM technique (Golyanik et al. , 2017)
• Non Rigid Structure from Motion (NRSfM)- input: point tracks on 2D frames- basically no limitation regarding target objects- multiple frames are required
Related works
![Page 10: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/10.jpg)
10Figure1. NRSfM technique (Golyanik et al. , 2017)
• Non Rigid Structure from Motion (NRSfM)- input: point tracks on 2D frames- basically no limitation regarding target objects- multiple frames are required- difficulty to apply on non-textured objects
Related works
![Page 11: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/11.jpg)
11
Related works
![Page 12: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/12.jpg)
12
Figure2. 3D reconstruction from a sequence of images (Yu et al,2015)
• Template based
Related works
![Page 13: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/13.jpg)
13
Figure2. 3D reconstruction from a sequence of images (Yu et al,2015)
• Template based- input: 3D template & 2D images
Related works
![Page 14: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/14.jpg)
14
Related works• Neural network based
![Page 15: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/15.jpg)
15
Related works• Neural network based
- Input: a single/sequence of images
![Page 16: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/16.jpg)
16
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)
![Page 17: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/17.jpg)
17
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)- E.g. HDM-Net
![Page 18: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/18.jpg)
18
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)- E.g. HDM-Net
Figure3. HDM-Net (Golyanik, 2018)
![Page 19: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/19.jpg)
19
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)- E.g. HDM-Net
- 3D Reconstruction from a single RGB imageFigure3. HDM-Net (Golyanik, 2018)
![Page 20: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/20.jpg)
20
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)- E.g. HDM-Net
- 3D Reconstruction from a single RGB image- Regress 3D coordinates (xyz geometry)
Figure3. HDM-Net (Golyanik, 2018)
![Page 21: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/21.jpg)
21
Related works• Neural network based
- Input: a single/sequence of images- Output: 3D geometry (Voxel/Point Set/Mesh)- E.g. HDM-Net
- 3D Reconstruction from a single RGB image- Regress 3D coordinates (xyz geometry)- Apply 2D conv. not 3D conv.
Figure3. HDM-Net (Golyanik, 2018)
![Page 22: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/22.jpg)
22
Limitations of HDM-Net
Template-based(Yu,2015)
Ours GT
![Page 23: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/23.jpg)
23
Limitations of HDM-Net
Template-based(Yu,2015)
Ours GT
• In a real-world scenario, our architecture has difficulty to reconstruct geometries especially when
![Page 24: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/24.jpg)
24
Limitations of HDM-Net
Template-based(Yu,2015)
Ours GT
• In a real-world scenario, our architecture has difficulty to reconstruct geometries especially when
1) the deformation states in the scene is quite different from the ones in the training dataset
![Page 25: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/25.jpg)
25
Limitations of HDM-Net
Template-based(Yu,2015)
Ours GT
• In a real-world scenario, our architecture has difficulty to reconstruct geometries especially when
1) the deformation states in the scene is quite different from the ones in the training dataset
2) the scene has a complicated background
![Page 26: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/26.jpg)
Overview
26
![Page 27: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/27.jpg)
Overview
27
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 28: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/28.jpg)
Overview
28
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 29: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/29.jpg)
Overview
29
![Page 30: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/30.jpg)
Overview
30
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 31: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/31.jpg)
Overview
31
OD-Net
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 32: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/32.jpg)
Overview
32
OD-Net
Confidence Map
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 33: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/33.jpg)
Overview
33
OD-Net
Confidence Map Binary Mask
Binarisation
Contour fill&
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 34: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/34.jpg)
Overview
34
OD-Net
Confidence Map Binary Mask
Binarisation
Contour fill&
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 35: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/35.jpg)
Overview
35
OD-Net
Confidence Map Binary Mask
Binarisation
Contour fill&
Masked-out Input
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 36: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/36.jpg)
Overview
36
OD-Net
Rec-Net
Confidence Map Binary Mask
Binarisation
Contour fill&
Masked-out Input
IsMo-GAN (Point Cloud Generator)Input
2D Image
![Page 37: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/37.jpg)
Overview
37
OD-Net
Rec-Net
Confidence Map Binary Mask
Binarisation
Contour fill&
Masked-out Input
IsMo-GAN (Point Cloud Generator)Output
3D Geometry
Input2D Image
![Page 38: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/38.jpg)
Overview
38
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 39: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/39.jpg)
Overview
39
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 40: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/40.jpg)
40
Loss functions
![Page 41: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/41.jpg)
• In order to penalize the network output critically, three kinds of loss functions were incorporated.
41
Loss functions
![Page 42: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/42.jpg)
1) 3D error
• In order to penalize the network output critically, three kinds of loss functions were incorporated.
2) Isometry prior 3) Adversarial loss42
Loss functions
Discriminator 1/0
![Page 43: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/43.jpg)
1) 3D error
43
Loss functions
![Page 44: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/44.jpg)
1) 3D error
• Main loss component for 3D coordinates regression
44
Loss functions
![Page 45: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/45.jpg)
1) 3D error
• Main loss component for 3D coordinates regression• Penalize difference between 3D coordinates of output and GT
45
Loss functions
![Page 46: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/46.jpg)
2) Isometry prior
46
Loss functions
![Page 47: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/47.jpg)
2) Isometry prior
• Idea: since we assume our target object is isometric, a vertex position has to be close to neighboring vertices.
47
Loss functions
![Page 48: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/48.jpg)
2) Isometry prior
• Idea: since we assume our target object is isometric, a vertex position has to be close to neighboring vertices.
48
Loss functions
![Page 49: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/49.jpg)
2) Isometry prior
• Idea: since we assume our target object is isometric, a vertex position has to be close to neighboring vertices.
49
Loss functions
![Page 50: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/50.jpg)
2) Isometry prior
• Idea: since we assume our target object is isometric, a vertex position has to be close to neighboring vertices.
• Apply gaussian smoothing on the output and compute the difference between XG and X.
50
Loss functions
![Page 51: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/51.jpg)
2) Isometry prior
• Idea: since we assume our target object is isometric, a vertex position has to be close to neighboring vertices.
• Apply gaussian smoothing on the output and compute the difference between XG and X.
51
Loss functions
![Page 52: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/52.jpg)
52
Loss functions
3) Adversarial loss
![Page 53: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/53.jpg)
53
Loss functions
3) Adversarial loss
Discriminator [0,1]Generator
Ground Truth
![Page 54: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/54.jpg)
54
Loss functions
3) Adversarial loss
• For further generalisability, the network is trained in an adversarial manner
Discriminator [0,1]Generator
Ground Truth
![Page 55: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/55.jpg)
Overview
55
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 56: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/56.jpg)
Overview
56
Point CloudGenerator
3D Output
GT
Input2D Image
Discriminator
![Page 57: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/57.jpg)
57
Datasets
![Page 58: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/58.jpg)
• Generated 4648 deformation states on blender game engine
state0 state1282 state3426 state4648
・・・ ・・・ ・・・
58
Datasets
![Page 59: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/59.jpg)
• Generated 4648 deformation states on blender game engine• Took 5 images for each state.
state0 state1282 state3426 state4648
・・・ ・・・ ・・・
59
Datasets
![Page 60: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/60.jpg)
• Generated 4648 deformation states on blender game engine• Took 5 images for each state.
• 4 different textures(Organ, Flag, Cloth, Carpet)
state0 state1282 state3426 state4648
・・・ ・・・ ・・・
60
Datasets
![Page 61: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/61.jpg)
• Generated 4648 deformation states on blender game engine• Took 5 images for each state.
• 4 different textures(Organ, Flag, Cloth, Carpet)• 5 different illumination positions
state0 state1282 state3426 state4648
・・・ ・・・ ・・・
61
Datasets
![Page 62: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/62.jpg)
• Generated 4648 deformation states on blender game engine• Took 5 images for each state.
• 4 different textures(Organ, Flag, Cloth, Carpet)• 5 different illumination positions• 4648 ply file and 330K images in total(4648x5x4x3 + 4648x5 + 4648x5)
state0 state1282 state3426 state4648
・・・ ・・・ ・・・
62
Datasets
![Page 63: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/63.jpg)
63
Evaluation and Visualization
![Page 64: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/64.jpg)
64
Template-based(Yu,2015) Ours GT
Quantitative Results
![Page 65: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/65.jpg)
65
Different textures
Template-based(Yu,2015) Ours GT
![Page 66: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/66.jpg)
66
Real-world images
Template-based(Yu,2015) Ours GT
(OD-Net)
(IsMo-GAN)
![Page 67: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/67.jpg)
67
Template-based(Yu,2015) Ours GT
Origami sequences
![Page 68: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/68.jpg)
68
Template-based(Yu,2015) Ours GT
Real texture-less clothTexture less dataset (Bednarik et al. , 2018)
![Page 70: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/70.jpg)
Conclusion
![Page 71: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/71.jpg)
Conclusion
• IsMo-GAN excels other model-based approaches in accuracy and inference time (250 HZ)
![Page 72: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/72.jpg)
Conclusion
• IsMo-GAN excels other model-based approaches in accuracy and inference time (250 HZ)
• Robust to illumination position changes
![Page 73: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/73.jpg)
Conclusion
• IsMo-GAN excels other model-based approaches in accuracy and inference time (250 HZ)
• Robust to illumination position changes• Thanks to OD-Net, IsMo-GAN shows better generalizability in a
texture-less and real-world scenario comparing with HDM-Net
![Page 74: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/74.jpg)
Conclusion
• IsMo-GAN excels other model-based approaches in accuracy and inference time (250 HZ).
• Robust to illumination position changes.• Thanks to OD-Net, IsMo-GAN shows better generalizability in a
texture-less and real-world scenario comparing with HDM-Net.• (Limitation) Training data (deformation state) is limited.
![Page 75: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/75.jpg)
75
Template-based(Yu,2015) Ours GT
References
1. Bednarik, J., & Fua, P., & Salzmann, M. (2018). Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View. In International Conference on 3D Vision (pp. 606-615).
2. Garg, R., Roussos, A., & Agapito, L. (2013). Dense variational reconstruction of non-rigid surfaces from monocular video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1272-1279).
3. Golyanik, V., Shimada, S., Varanasi, K., & Stricker, D. (2018, October). HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model. In International Conference on Virtual Reality and Augmented Reality (pp. 51-72). Springer, Cham.
4. Golyanik, V., & Stricker, D. (2017, March). Dense batch non-rigid structure from motion in a second. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 254-263). IEEE.
5. Liu-Yin, Q., Yu, R., Agapito, L., Fitzgibbon, A., & Russell, C. (2016). Better together: Joint reasoning for non-rigid 3d reconstruction with specularities and shading. In Proceedings of the British Machine Vision Conference (pp. 42.1-42.12.).
6. Yu, R., Russell, C., Campbell, N. D., & Agapito, L. (2015). Direct, dense, and deformable: Template-based non-rigid 3d reconstruction from rgb video. In Proceedings of the IEEE International Conference on Computer Vision (pp. 918-926).
![Page 76: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/76.jpg)
76
Template-based(Yu,2015) Ours GT
Thank you for your attention!
![Page 77: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/77.jpg)
Texture-less dataset
![Page 78: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/78.jpg)
• Extract 20 sequential deformation from each 100 states as a test dataset
• Training:Test = 8 : 2
78
Datasets
![Page 79: IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D ...gvv.mpi-inf.mpg.de/projects/IsMO-GAN/data/ISMO-GAN_PCV_CVPRW.pdfTemplate-based(Yu,2015) Ours GT •In a real-world scenario,](https://reader034.fdocuments.us/reader034/viewer/2022042605/5f53a8081367037a8d605036/html5/thumbnails/79.jpg)
79
External Occlusion
Template-based(Yu,2015) Ours GT