Heterogeneous Computing in 3D Perception Chris Slaughter, CEO Jeff Mahler, Computer Vision

Architecture

Video Games / AR

Visual Effects

Robotics

Architecture: Convenient 3D capture from a moving camera

Video Games / AR: Texture capture, live feed augmentation

Robotics: Capture, semantic interpretation, perception for manipulation

Visual Effects: Fast 3D capture for compositing, animation, lighting effects

All of these problems are

3D Perception. Perception - awareness of the elements of environment through physical sensation

3D Perception - awareness of the elements of environment through physical sensation in 3D

Input Output

3D perception tasks can be done in

Real Time.

3D Rendering

Application Scene Cull Draw Rendering

3D Perception

Cognition Scene Understanding

Infer Estimate Sensors

Today’s Agenda

Techniques • Motion Tracking • 3D Capture • Global Refinement • 3D Image Processing

Applications •Bomb Detection •Lynx A Camera

Future of Heterogeneous Computing!

Motion Tracking

Data Source Representation

Data Correspondence

Error Minimization

Objective Linearization A

x

b

-

||Ax – b||2

Normal Equations Parallel Reduction

A

x

b

- AT AT

||ATAx – ATb||2

Color

Position

Curvature

Orientation

3D Capture

Data Source Representation

Tracking

Refinement

Techniques

System Representation Tracking Refinement

KinectFusion1 Volumetric Projective Point-

Plane ICP Truncated Signed

Distance

Dense Tracking and Mapping2 Volumetric / 2.5D

Lucas-Kanade Alignment

None

Patchwork3 2.5D Projective Point-

Plane ICP Kernelized

Stochastic Descent

1. Richard Newcombe / Microsoft; 2. Richard Newcombe; UT Austin & Lynx Labs

Tracking

Refinement

Texture Reads (4x)

Surface Writes (1x)

Global Refinement

Raw Data Post Processing

Trajectory Pose Graph

Reprojection Error

Fast Data Correspondences

Sparsity in Adjacency Matrix

3D Image Processing

Anisotropic Diffusion

Fast Digital Inpainting

Harris Corner Detection

λ1 indicates the presence of an edge in the x-direction

λ2 indicates the presence of an edge in the y-direction

Bomb Detection

Pre-Captured 3D Model

Live Dense 3D Feed

- =

No Bomb Bomb

Lynx A Camera

Conclusion

• Many emerging applications for 3D perception

• Heterogeneous computing enables real-time applications

• GPU architecture is ideal for 3D

perception processing