Autonomous Vision Group

Max Planck Institutefor Intelligent Systems

Texture FieldsLearning Texture Representations in Function Space

Michael Oechsle1,2,3 Lars Mescheder1,2 Michael Niemeyer1,2

Thilo Strauss3 Andreas Geiger1,2

¹MPI for Intelligent Systems 2University of Tübingen 3ETAS GmbH, Stuttgart

michael.oechsle@tue.mpg.de

1. Motivation

Existing 3D Representations

Deep learning has achieved impressive results •

for image and texture generation in the 2D domain

However, texture generation in the 3D domain lacks far behind.

Major problem: Representation of texture in 3D

for learning-based reconstruction of 3D geometry.•

Colored Voxels

Grid structure

Low-frequency texture

Memory requirement

Texture Atlas

Texture from a Single Rendered Image

Texture from a Single Real Image

Quantitative Evaluation

2. Existing Texture Representations

No discretization

Independent ofshape representation

No template required

For learning texture reconstruction, we condition the Texture Field on an image and an untextured 3D model.

Idea: Represent texture as continuous 3D field

3. Our Representation

2D Image Ours ProjectionNVS baseline

2D Image 2D ImageOurs Ours

2D Image ONet ONet + Texture Fields

5. Experiments - Texture Reconstruction

4. Our Model

High-frequency texture

Discontinuities

Template required

Textured 3D Model

3D Model

2D Image

Neural Network

6. Experiments - Generative Model

Full Pipeline

Shape

Texture

VAE - Texture TransferGAN - SamplesVAE - Samples

VAE - Latent Space Interpolations

Texture from a Single Rendered Image

Full Pipeline with ONet

tiny.cc/texfield

with slides, videos and more

Blog

KL Divergence

Shape Encoder

Texture Field

Sampling

Image Encoder

3D Shape

2D Image

Recon.Loss

Depth Map 3D Point Predicted Image True Image

Point Cloud

Rendering

Unprojection

VAE Encoder

Adver-sarialLossConditional Model

GAN ModelVAE Model

Color Legend:

Color

Discriminator