Iuri Frosio, GTC 2019 (San Jose, CA)

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

2

THE IMPORTANCE OF NEGATIVE RESULTS

“I shall require that [the] logical form [of thetheory] shall be such that it can be singledout, by means of empirical tests, in anegative sense: it must be possible for anempirical scientific system to be refuted byexperience” (Karl Popper, The Logic ofScientific Discovery, 1959).

In simple words, “negative results arefundamentals for the advancement ofscience”.

3

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

4

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTSE.g., Kinect 2

Image from https://stackoverflow.com/questions/22921390/how-to-scale-a-kinect-

depth-image-for-applying-lbp-on-it-in-matlab?rq=1

5

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTSApplications

Image from https://www.physio-

pedia.com/The_emerging_role_of_Microsoft_Kinect_in_physiotherapy_rehabilitation_for_stroke_patients

Image from amazon.com

Image from https://www.eenewsautomotive.com/news/3d-lidar-generates-environmental-model-time-flight-

measurement

6

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTSArtifact #1: shot noise

Image from https://ptgrey.com

7

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTSArtifact #2: movement

Image from https://support.xbox.com

8

TIME OF FLIGHT (TIF) CAMERAS & ARTIFACTSArtifact #3: multiple reflections

Image from

https://www.sciencedirect.com/science/article/pii/S0262885613001650

9

TIME OF FLIGHT (TIF) CAMERAS & ARTIFACTSArtifact #3: multiple reflections

10

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

11

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

What do we need?

(1) A large dataset of scenes…

(2) … corrupted by:

(1.1) photon noise,(1.2) motion,(1.3) multiple reflections…

(3) … with clean output data…

(4) … And a DNN.

12

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

13

TOF WORKING PRINCIPLESTime of flight is not time of flight ☺

Image from https://www.semanticscholar.org/paper/Interference-mitigation-technique-for-(ToF)-camera-Islam-Hossain/

14

TOF WORKING PRINCIPLESTime of flight is not time of flight ☺

Images from https://www.semanticscholar.org/paper/Interference-mitigation-technique-for-(ToF)-camera-Islam-Hossain/

15

TOF WORKING PRINCIPLESMultiple measurements

Pulse method

Continuous wave method

16

TOF WORKING PRINCIPLESCamera functions and scene response

17

TOF WORKING PRINCIPLEMore on scene response

= න−𝑇2

𝑇2𝑓 𝑡 ⨂𝑔𝑖 𝑡 𝑟 𝑡 𝑑𝑡= න

−𝑇2

𝑇2𝑓 𝑡 ∗ 𝑟 𝑡 𝑔𝑖 𝑡 𝑑𝑡Raw measurement:

Emitted signal: 𝑓 𝑡

Returned signal: ℎ 𝑡

Demodulation signal: 𝑔𝑖 𝑡

Impulse response: 𝑟(𝑡)

Camera Scene

𝑄𝑖(𝑡) = න−𝑇2

𝑇2ℎ 𝑡 𝑔𝑖 𝑡 𝑑𝑡

Depth: 𝑍 =𝑇𝑐

4𝜋arctan

σ𝑖 sin𝜃𝑖 𝑄𝑖 𝑡

σ𝑖 cos𝜃𝑖 𝑄𝑖 𝑡

t

t

tDemodulation: gi(𝑡)

18

TOF WORKING PRINCIPLESMultiple frequencies

• Different max length (combinethem)

• Different resolutions

• Agreement between differentmeasurements

19

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

20

CAMERA CALIBRATIONCamera response functions (flat scene)

Inside coated with black-out material

21

CAMERA CALIBRATIONCamera response functions (flat scene)

• Three “frequencies”

• Three measurements perfrequency

22

CAMERA CALIBRATIONPhoton noise

Other calibration details (pixel delay, vignetting, … in Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany), Sept. 2018.

23

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

24

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

To disk (FLAT)https://github.com/NVlabs/FLAT

25

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

Transient rendering (scene response function) based on Jarabo, A., Marco, J., Muñoz, A.,Buisan, R., Jarosz, W., Gutierrez, D.: A framework for transient rendering. In: ACMTransactions on Graphics (SIGGRAPH ASIA), (2014).

26

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

Transient rendering (scene response function) based on Jarabo, A., Marco, J., Muñoz, A.,Buisan, R., Jarosz, W., Gutierrez, D.: A framework for transient rendering. In: ACMTransactions on Graphics (SIGGRAPH ASIA), (2014).

Brightness, travel time

27

Impulse response

28

Impulse response

29

Impulse response

30

Impulse response

31

Impulse response

32

Impulse response

33

Impulse response

34

Impulse response

35

Impulse response

36

Impulse response

37

Impulse response

38

Impulse response

39

Impulse response

40

Impulse response

41

Impulse response

42

Impulse response

43

Impulse response

44

Impulse response

45

Impulse response

46

Impulse response

47

Impulse response

48

Impulse response

49

Impulse response

50

Impulse response

51

Impulse response

52

Impulse response

53

Impulse response

54

Impulse response

55

Impulse response

56

Impulse response

57

Impulse response

58

Impulse response

59

Impulse response

60

Impulse response

61

Impulse response

62

Impulse response

63

Impulse response

64

Impulse response

65

Impulse response

66

Impulse response

67

Impulse response

68

Impulse response

69

Impulse response

70

Impulse response

71

Impulse response

72

Impulse response

73

Impulse response

74

Impulse response

75

Impulse response

76

Impulse response

77

Impulse response

78

Impulse response

79

Impulse response

80

Impulse response

81

Impulse response

82

Impulse response

83

Impulse response

84

Impulse response

85

Impulse response

86

Impulse response

87

Impulse response

88

Impulse response

89

Impulse response

90

Impulse response

91

Impulse response

92

Impulse response

93

Impulse response

94

Impulse response

95

Impulse response

96

Impulse response

97

Impulse response

98

Impulse response

99

Impulse response

100

Impulse response

101

Impulse response

102

Impulse response

103

Impulse response

104

Impulse response

105

Impulse response

106

Impulse response

107

Impulse response

108

Impulse response

109

Impulse response

110

Impulse response

111

Impulse response

112

Impulse response

113

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

Different cameras (beyond Kinect 2) can be simulated, after calibration.

114

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

Noise can be added…

115

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

… As well as motion (approximate model) and texture…

116

THE FLAT DATASETFlexible, Large, Augmentable, ToF (FLAT)

… and multiple reflections.

117

THE FLAT DATASETSamples

118

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

119

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

Supervised learning

Take it easy: supervised learning, from raw data to 3D map.

Training input/output pairs from the FLAT dataset.

120

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

Supervised learning

121

THE LESSON WE LEARNED*…* To advance science.

That’s a nice negative result!

122

… AND HOW WE IMPROVED

*1 Yes, it’s a fakepicture…

*2 … But the message iscorrect.

*1

*2

123

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

124

BACK TO PHYSICS

Cause: Sequential measurements

Effect: Misaligned moving object

Solution:Warping

And, more generally speaking, any a-priori knowledge.

time

125

BACK TO PHYSICS

Cause: DNN architecture andlearningmocks physics

Effect: Sub-optimal results

Solution:Include physics in the DNNarchitecture / reconstructionpipeline.

And, more generally speaking, any a-priori knowledge.

Naïve machine learning

Physics

126

DNN ARCHITECTURE#1: Motion Correction Module

Trained to warp imagesto the central one

127

DNN ARCHITECTURE#2: Motion Reflection Module

Trained to reducemultiple reflection afterre-alignment.

128

DNN ARCHITECTURE#3: Differential reconstruction pipeline

Non-trainable, butdifferentiable,physics-based reconstructionpipeline.

129

DNN ARCHITECTURE#3: Differential reconstruction pipeline

Can be refined withend-to-end training.

130

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

131NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

RESULTS

Compare against

LF2 [1] [Kinect, non DL]

DToF [3] [DL, Raw to 3D, no motion]

Phasor [2] [High frequencies reduce MPI]

[1] Xiang, et al. libfreenect2: Release 0.2[2] Marco, et al. DeepToF: Off-the-shelf real -time correction of multipath interference in time-of-flight imaging. In: ACM Transactions on Graphics (SIGGRAPH ASIA).[3] Gupta, et al. Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics.

Competitors & ablation study

Ablation study

MOM [motion only]

MRM [multiple reflection and noise only]

MOM-MRM [motion, multiple reflection and noise ]

132

RESULTSAblation study: none, MRM, MOM+MRM [simulation]

Median [Med] and Inter Quartile Range [IQR] of the error decreased by MRM / MOM-MRM, in cm.

133

RESULTSAblation study: none, MRM, MOM+MRM [simulation]

134

RESULTSCompare against: DTOF, Phasor imaging [simulation]

Smaller error when compared to DToF or Phasor.

135

RESULTSCompare against: DTOF, Phasor imaging on multi-reflection and shot

noise [simulation]

Field of view

Multiple reflection removed through local reflection / a-priori information, no bias.

136

RESULTSCompare against: LF2, on multi-reflection and shot noise [real data]

Multiple reflection removed through local reflection / coherence / a-priori information, no bias.

137

RESULTSCompare against: LF2, on movement [real data]

Realignment of raw data reduce motion artifacts, specular reflections (red box) generate errors.

138

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

- Time Of Flight (TOF) cameras & artifacts

- Naïve Machine Learning (ML) for TOF reconstruction

- TOF cameras: working principles

- Camera calibration

- The FLAT dataset

- Spoiler: our non-Naïve ML solution works*

- Back to physics

- DNN architecture

- Results

- Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo,Todd Zickler, Jan Kautz, Tackling 3D ToFArtifacts Through Learning and the FLATDataset, ECCV 2018, Munich (Germany),Sept. 2018.

139

CONCLUSION

1. Naïve ML does not always work…

140

CONCLUSION

1. Naïve ML does not always work…

2. ….But going back to a priori knowledge may help.

141

CONCLUSION

The physics of ToF cameras: acquisition, reconstruction, artifacts

Photon shot noise, motion artifacts, multiple reflection

A large dataset of simulated data

Design the DNN architecture accordingly to a-priori knowledge

Effective reduction of reconstruction artifacts

142

RESOURCES

Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz; The European Conference on Computer Vision (ECCV), 2018, pp. 368-383, http://openaccess.thecvf.com/content_ECCV_2018/html/Qi_Guo_Tackling_3D_ToF_ECCV_2018_paper.html

The FLAT dataset (code and data): https://github.com/NVlabs/FLAT

Contact: {ifrosio, ogallo}@nvidia.com, qiguo@g.harvard.edu

Online