Copyright © 2017 1

3D Human Motion Capture

from 2D Video using

Cloud-Based CNNs

NVIDIA GPU Technology Conference

Copyright © 2017 2

Imagine if Cameras Could See People

Computer Vision

bodySLAM™ turns 2D video into 3D humans

2D video 3D human

Copyright © 2017 3

Applications: AR Social Imagery & Filters

Copyright © 2017 4

AR / VR Gaming

Copyright © 2017 5

Health & Wellness

Copyright © 2017 6

Sport Analytics

Copyright © 2017 7

Eyes for Virtual Assistants

Copyright © 2017 8

APIIn: 2D Video

fidelity/performance parameters

Out: Human FeaturesJoints: 21 joints including eyes and earsSkeletons: 2D, 3DSegmentation: background, individuals, limbs

PlatformsCUDAMetal Performance ShadersopenCL

BodySLAM™: Deep Learning API

Copyright © 2017 9

Deep Learning / Computer Vision Secret Sauce100+ layer CNN (Caffe)Synthetic Training dataHuman tracking

Game Tech3D animationJoint FilteringInverse Kinematics

PlatformsNVIDIA GPUAWSMPS iOS

Tech Stack

Copyright © 2017 10

High Level Architecture

Copyright © 2017 11

Training Pipeline

Copyright © 2017 12

BodyFilter™: VFX filters built on top of BodySLAM™

Copyright © 2017 13

Unity Game Engine for Training Pipeline and VFX App

Copyright © 2017 14

Live Demos

Copyright © 2017 15

Working with CustomersOptimizationNative Mobile SupportHigher Fidelity

Future Work

Copyright © 2017 16

Questions? Let’s Start Augmenting Reality Together!