government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Deep 3D – Machine Learning for Reconstruction and Repair

of 3D SurfacesTalkID 23152

This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep learning techniques and an insightinto our approach for 3D surface repair.

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

• PhD Student at the Institute for Optical Systems at the HTWG Konstanz

• Main focus: Machine Learning for…• … Surface Reconstruction• … Defect Detection and Repair (Inpainting)• … Medical Imaging

pascal.laube@gmail.com

Pascal Laube

government-funded by supervised by

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representation: The 2D case

Output

Grid in euclidean space Neural Network(in this case CNN)

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representation: In 3D?

Neural Network

Point Cloud

Mesh

Any manifold (NURBS, impl. surf., …)

?

?

?

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Voxels

[Vishakh Hegde et al., NIPS (2016)]

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Voxels

[Zhirong Wu et al., CVPR (2015)]

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Multi-View

[Hang Su et al., ICCV (2015)]

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Multi-View

[Liuhao Ge et al., CVPR (2016)]

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Graph Signal Processing

[M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]

• Graph Laplacian or Laplace Beltrami Operator as

∆𝑓 = −𝑑𝑖𝑣(𝛻𝑓)

• Laplacian Eigenfunctionsgeneralize to Fourier bases.

Convolution in the spectral domain is defined…

…but filters are base dependent.

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Graph Signal Processing

• Train filters in geodesic polar coordinates.

• Pool rotation angles

[J. Masci et al., ICCV (2015)]• Many other methods using

different kernels (heat diffusion, gauss…)

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Data Sets

• 127,915 CAD Models• 662 Object Categories• Different Subsets

• 51,300 Models• 270 Object Categories in 12.000 Model Subsets

Many smaller specialized Data Sets

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry

Fraunhofer IPT

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry

Werkzeugbau Siegfried Hofmann GmbH

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry (3)

• High resolution meshes with > 1m vertices

• Base Geometry and Relief

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Our Approach

B-Spline Approx.

Approx. by Geometric Primitive

Multiresolut. Surfaces

SeperationBase Geo. – Detail Geo.

Surface with Defect Novelty Detection usingAutoencoders

Multiresolution Neural Netsfor Inpainting

Detail GeometryHeightmap

Base Geometry

1 2 3

or

or

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

• Defect unknown• Healthy state unknown

What do we know?

• Textures have to be ergodic:Statistical properties are constant for single sample and whole collection

2

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

Train Autoencoder on Ergodic Set of Textures

Autoencoder should be unableto sufficiently reconstruct Defects

2

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

Loss

Samples

2

Parallelizable to multiple GPUs

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Texture Synthesis3

[L. Gatys et al., NIPS (2015)]

Activation Network Synth. Network

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3

[L. Gatys et al., arxiv.org (2015)]

Multiresolution Neural Nets for Inpainting: Style Transfer

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Example3

2048x2048

Defect

Closeup

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Patches3

• Inpainting a Region with arbitrary size?• Inpaint Patch by Patch

Local Style Global Style

2048x2048

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Results3

1. Start

2. Inpaint Patches:• Large Parent Weight

…

3. Inpaint Patches:• Apply Detail• Large Child Weight• Small Parent Weight

…

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Results3

Result Result Closeup

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3 Multiresolution Neural Nets for Inpainting: Results Heightmap

Parallelizable to multiple GPUs

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3 Multiresolution Neural Nets for Inpainting: Results Surface

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Outlook

[J. Masci et al., ICCV (2015)]

[M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]

• Neural Nets in high dimensional irregular domains

• Michael M. Bronstein et al., “Geometric deep learning: going beyond Euclidean data” (2017)

• Michaël Defferrard, “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering” (2016)

government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Thank You