Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous...
Transcript of Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous...
![Page 1: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/1.jpg)
Heterogeneous Computing in 3D Perception Chris Slaughter, CEO Jeff Mahler, Computer Vision
![Page 2: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/2.jpg)
Architecture
![Page 3: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/3.jpg)
Video Games / AR
![Page 4: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/4.jpg)
Visual Effects
![Page 5: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/5.jpg)
Robotics
![Page 6: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/6.jpg)
Architecture: Convenient 3D capture from a moving camera
Video Games / AR: Texture capture, live feed augmentation
Robotics: Capture, semantic interpretation, perception for manipulation
Visual Effects: Fast 3D capture for compositing, animation, lighting effects
![Page 7: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/7.jpg)
All of these problems are
3D Perception. Perception - awareness of the elements of environment through physical sensation
3D Perception - awareness of the elements of environment through physical sensation in 3D
![Page 8: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/8.jpg)
Input Output
![Page 9: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/9.jpg)
3D perception tasks can be done in
Real Time.
![Page 10: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/10.jpg)
3D Rendering
Application Scene Cull Draw Rendering
![Page 11: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/11.jpg)
3D Perception
Cognition Scene Understanding
Infer Estimate Sensors
![Page 12: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/12.jpg)
Today’s Agenda
Techniques • Motion Tracking • 3D Capture • Global Refinement • 3D Image Processing
Applications •Bomb Detection •Lynx A Camera
Future of Heterogeneous Computing!
![Page 13: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/13.jpg)
Motion Tracking
![Page 14: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/14.jpg)
Data Source Representation
![Page 15: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/15.jpg)
Data Correspondence
Error Minimization
![Page 16: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/16.jpg)
Objective Linearization A
x
b
-
||Ax – b||2
![Page 17: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/17.jpg)
Normal Equations Parallel Reduction
A
x
b
- AT AT
||ATAx – ATb||2
![Page 18: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/18.jpg)
Color
Position
Curvature
Orientation
![Page 19: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/19.jpg)
![Page 20: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/20.jpg)
3D Capture
![Page 21: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/21.jpg)
Data Source Representation
![Page 22: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/22.jpg)
Tracking
Refinement
![Page 23: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/23.jpg)
Techniques
System Representation Tracking Refinement
KinectFusion1 Volumetric Projective Point-
Plane ICP Truncated Signed
Distance
Dense Tracking and Mapping2 Volumetric / 2.5D
Lucas-Kanade Alignment
None
Patchwork3 2.5D Projective Point-
Plane ICP Kernelized
Stochastic Descent
1. Richard Newcombe / Microsoft; 2. Richard Newcombe; UT Austin & Lynx Labs
![Page 24: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/24.jpg)
Tracking
Refinement
Texture Reads (4x)
Surface Writes (1x)
![Page 25: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/25.jpg)
![Page 26: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/26.jpg)
![Page 27: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/27.jpg)
![Page 28: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/28.jpg)
![Page 29: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/29.jpg)
![Page 30: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/30.jpg)
Global Refinement
![Page 31: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/31.jpg)
Raw Data Post Processing
![Page 32: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/32.jpg)
Trajectory Pose Graph
![Page 33: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/33.jpg)
Reprojection Error
![Page 34: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/34.jpg)
Fast Data Correspondences
Sparsity in Adjacency Matrix
![Page 35: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/35.jpg)
![Page 36: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/36.jpg)
3D Image Processing
![Page 37: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/37.jpg)
Anisotropic Diffusion
![Page 38: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/38.jpg)
![Page 39: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/39.jpg)
Fast Digital Inpainting
![Page 40: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/40.jpg)
![Page 41: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/41.jpg)
Harris Corner Detection
λ1 indicates the presence of an edge in the x-direction
λ2 indicates the presence of an edge in the y-direction
![Page 42: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/42.jpg)
![Page 43: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/43.jpg)
Bomb Detection
![Page 44: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/44.jpg)
Pre-Captured 3D Model
Live Dense 3D Feed
![Page 45: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/45.jpg)
- =
![Page 46: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/46.jpg)
No Bomb Bomb
![Page 47: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/47.jpg)
![Page 48: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/48.jpg)
Lynx A Camera
![Page 49: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/49.jpg)
![Page 50: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/50.jpg)
![Page 51: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/51.jpg)
![Page 52: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/52.jpg)
Conclusion
• Many emerging applications for 3D perception
• Heterogeneous computing enables real-time applications
• GPU architecture is ideal for 3D
perception processing
![Page 53: Heterogeneous Computing in 3D Perception | GTC 2013 · 2013. 3. 20. · Title: Heterogeneous Computing in 3D Perception | GTC 2013 Author: Christopher Slaughter, Jeff Mahler Subject:](https://reader035.fdocuments.us/reader035/viewer/2022063022/5fe8d7ef10c1c94f170e9657/html5/thumbnails/53.jpg)