Kinect Fusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera

1LEAAICHNER,1226600

SHAHRAM IZADI, DAVID KIM, OTMAR HILLIGES, DAVID MOLYNEAUX, RICHARD NEWCOMBE, PUSHMEET KOHLI, JAMIE SHOTTON, STEVE HODGES, DUSTIN FREEMAN, ANDREW DAVIDSON, ANDREW FITZGIBBON

PROCEEDINGS OF THE 24TH ANNUAL ACM SYMPOSIUM ON USER INTERFACE SOFTWARE AND TECHNOLOGY. ACM, 2011.

Overview

2

 Difficultgoal 3Dreconstructionofanindoorscene Usesingledepthcamera◦  Estimateposeofcamera◦  Comparedepthmap◦  Update3Dreconstruction

 Low-costandreal-time RelatedWork:◦  Activesensors◦  Passivecameras◦  OnlineImages◦  SimultaneousLocalizationandMapping(SLAM)

LEAAICHNER,1226600

Design Goals  Interactiveratesforcameratrackingandreconstruction◦ Directfeedback◦ Userinteraction

 Noexplicitfeaturedetection◦ Cameratrackingavoidsexplicitdetectionstep◦ Worksondepthmaps

 High-qualityreconstructionofgeometry

3LEAAICHNER,1226600

Design Goals  Dynamicinteractionassumed◦ userinteractionispossible◦ Dynamicallychangingscenes

 Infrastructure-less◦ Reconstructarbitraryindoorspaces

 Roomscale◦ Supportroomreconstructionsandinteraction

4LEAAICHNER,1226600

KinectFusion System  Construct3Dmodelofthescene:◦ Track6DOFposeofcamera◦ Fuselivedepthdataintoa3Dmodel

 Userexploresthespace◦ Newviews◦ Reconstructiongrows◦  Imagesuper-resolution

5LEAAICHNER,1226600

Examples

6LEAAICHNER,1226600

Object Segmentation  Scanspecificphysicalobject◦ Monitor3Dreconstruction◦ Observechangesovertime◦ Segmentrepositionedobject

7LEAAICHNER,1226600

Geometry-Aware Augmented Reality  3Dvirtualworldisoverlaidontotherealworld

8LEAAICHNER,1226600

Taking Physics Beyond the Surface  Simulatereal-worldphysics.

9LEAAICHNER,1226600

Reaching into the Scene  Userinteraction◦ Staticscene->dynamicscene◦ Robusttotransientandrapidscenemotions◦ Problemswithprolongedinteractions◦  Usermovesinfrontofthecamera

 SpecialGPU-basedpipeline◦ Geometryofbackgroundscene◦ Geometryoftheforegrounduser

 Determineinteractions

10LEAAICHNER,1226600

System pipeline

11LEAAICHNER,1226600

Camera Tracking  IterativeClosestPoint(ICP)◦  Projectivedataassociation◦  Findcorrespondencesbetweenorientedpoints

 Output:relativetransformationmatrixthatminimizesthepoint-to-planeerrormetric Densetracking

12

D:DepthmapT:globalcameraposeV:vertexmapN:NormalmapR:Rotationmatrix

LEAAICHNER,1226600

Volumetric Representation  3Dvolumewithfixedresolution Integrate3DverticesintovoxelsusingSignedDistanceFunction(SDF)◦ Surfacedefinedbythezero-crossing

 TruncatedSignedDistanceFunction(TSDF)

 3DvoxelgridisallocatedontheGPUasalignedlinearmemory

13LEAAICHNER,1226600

Summary

 Features:◦ NovelGPUpipeline–realtime◦ Low–costobjectscanning◦ Physicsbasedinteraction◦ Dynamiccontent

14

 Futurework◦ Reconstructionoflargerscenes◦ Moredetailsinthereconstruction◦ Opennewresearchtopics

LEAAICHNER,1226600

 3Dreconstructionandcameraposeestimationusingsingledepthcamera

References  1.S.Izadietal.,“KinectFusion:real-time3Dreconstructionandinteractionusingamovingdepthcamera,”inProceedingsofthe24thannualACMsymposiumonUserinterfacesoftwareandtechnology,2011,pp.559–568.

 2.https://msdn.microsoft.com/en-us/library/dn188670.aspx

15LEAAICHNER,1226600