RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf ·...
Transcript of RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf ·...
![Page 1: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/1.jpg)
RGBD Image Processing for 3D Modeling and Texturing
ICIP 2013 Tutorial Dr. Hwasup Lim, Dr. Seungkyu Lee,
and Dr. Jongwoo Lim
1
![Page 2: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/2.jpg)
Commodity Depth Cameras Product Name
Product Image
Depth Sensing
Resolution Distance Range
Frame Rate
Price
PrimeSense Carmine
Structured Light
D640x480 C640x480
0.8m-3.5m 30fps $200
MS Kinect for Windows
Structured Light
D640x480 C640x480
0.8m-4m (0.4m-3.0m
) 30fps $250
Asus Xtion Pro Live
Structured Light
D640x480 C640x480
0.8m-3.5m 30fps $200
Intel/Creative Senz3D
Time of Flight
D320x240 C1280x720
0.15m-1m 30fps(D) $149
Softkinectic DS325
Time of Flight
D320x240 C1280x720
0.15m-1m 30fps(D) $249
MS Kinect 2.0 for Xbox One
Time of Flight
D512×424 C1920x108
0 ? 30fps ?
2
![Page 3: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/3.jpg)
RGBD Paper Statistics
3
[Google Scholar]
Keyword “RGBD”
Keywords “Kinect + simultaneous localization and mapping, object reconstruction, multiple Kinect, interference mitigation, transparency and calibration”
![Page 4: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/4.jpg)
3D Imaging Technology
Extract or Reconstruct 3D Information from One or Multiple Imaging
Sensors.
- 2D Imaging : Intensity of incoming light ray
- 3D Imaging : Intensity of all light rays of the light field
Lambertian Surface Assumption has been made for 3D Imaging
Non-Lambertian Surface Extension is required for real 3D Imaging
,,,,,, tzyxPI
),( 11
),( 22
1
2
t
),,( zyx
Light Field
![Page 5: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/5.jpg)
Structured Light
- Project light patterns and detect correspondence
- Perform a triangulation between matching points
Pros: Low price high accuracy, High resolution
Cons: Limitation in sensing range (50cm~2m) and dynamic imaging, Occlusion
![Page 6: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/6.jpg)
Kinect Principle (1/4)
Basically, it is based on structured light principle
IR Speckle
Pattern
![Page 7: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/7.jpg)
Kinect Principle (2/4)
0. Calibrate source and
detector
1. Known IR pattern is
projected from the source
2. Detector identify each
dot (or set of dots)
3. Triangulate to calculate
depth
![Page 8: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/8.jpg)
Kinect Principle (3/4)
- Random speckles identify x,y locations
- Orientation and shape of the speckles change along distance
identify z location
![Page 9: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/9.jpg)
Kinect Principle (4/4)
Measured Depth and Color Images
![Page 10: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/10.jpg)
Time-of-Flight Principle (1/6)
Reflected IR shows phase delay proportional to the distance
from the camera.
Time-of-flight of Light Distance
: It is not simple to measure the flight time directly at each pixel
of any existing image sensor
![Page 11: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/11.jpg)
Laser Scanning
- Structured Light, Time-of-Flight principles are applied with Laser Light
Pros: High accuracy and repeatability, Long sensing range
Cons: Active sensing using Laser beam, Limitation in On-line 3D sensing
Structured Laser Light
LIDAR (Time-of-Flight)
![Page 12: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/12.jpg)
Time-of-Flight Principle (2/6)
Q1 through Q4 are the amount of electrons measured at each
corresponding time.
In real situations, it is difficult to sense electric charge at certain
time instance
![Page 13: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/13.jpg)
Distance Measurement
![Page 14: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/14.jpg)
IR Modulation
irTOF
irTOFon
inTN
iTTnN
1
0
10
1
1
0 NN
NTT
T
TT
N
NonTOF
TOF
TOFon
TX0
TX1
Emitted
Light
Reflected
Light
Ton
TTOF
N0/n
N1/n
TX0
TX1
Emitted
Light
Reflected
Light
TTOF
N0/n
N1/n
2cos2sin
2cos2sin
2
1
00
TOFTOF
TOFTOF
TadaTaN
TadaTaN
10
1
1
0
2cos
2cos
NN
NTT
T
T
N
NonTOF
TOF
TOF
Pulse Light
Sinusoidal Light
![Page 15: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/15.jpg)
Demodulation-related Error
Depth Calculation Error
Gate1
Gate2
Emitted NIR
Reflected NIR
Gate3
Gate4
A0
T0
td r·A0
Q1 Q1
Q2 Q2
Q3 Q3
Q4 Q4
td
Y1≥0, Y2≥0 Y1<0, Y2≥0 Y1<0, Y2<0 Y1≥0, Y2<0
0 T0/2 T0 3T0/2 2T0
Y
─ Y1 = nQ1-nQ2
─ Y2 = nQ3-nQ4
1
2d
Y
Yarctant
Four Phase depth calculation Difference in depth calculation
![Page 16: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/16.jpg)
Time-of-Flight Principle (3/6)
Distance
21
43arctan2
)(2 QQ
QQcdt
c
21
43
21
43 arctan2
arctan2 qq
qqc
qqc
Assumption: Single reflected IR signal
In principle, amplitude of the reflected IR does not affect the depth
calculation.
![Page 17: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/17.jpg)
Time-of-Flight Principle (4/6)
- Large Sensor Pixel
- Scattering
- Multipath
- Motion Blur
- Transparent Object
In real situations, multiple reflected IR signals with different phase
delays & amplitudes can be superposed.
)()(
)()(arctan
2)(
2211
4433
qqqq
qqqqcdt
We do not know how many IR signals will be superposed.
![Page 18: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/18.jpg)
Multiple Light Reception: Due to the interference of multiple IR light reflections Use multipath interference model [5]
Multipath Errors
IR LED
Sensor
Multipath Interference
![Page 19: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/19.jpg)
2-Layer approximation of transparent object
))1(())1((
))1(())1((arctan
2)(
2211
4433
qqqq
qqqqcdt
- Sometimes 2-Layer is not enough
- Multiple reflection between objects (when they are close)
- In most cases, they have specular surface
Multipath Errors (Transparency)
![Page 20: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/20.jpg)
Depth error in transparent objects
Multipath Errors (Transparency)
)(
)(arctan
21
43
QQtd
)ˆˆ()(
)ˆˆ()(arctan
2121
4343
QQQQ
QQQQtd
Depth
IR-Intensity
![Page 21: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/21.jpg)
Time-of-Flight Principle (5/6)
What we have calculated is the distance R from camera to
an object surface along the reflected IR.
Based on the location of each pixel and field of view information,
Z can be calculated from R to obtain undistorted 3-dimensional
geometry.
![Page 22: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/22.jpg)
Time-of-Flight Principle (6/6)
Measured Depth and IR Intensity Images
![Page 23: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/23.jpg)
Caused by irregularity (asymmetry) of the modulation process
Demodulation Error
PMD, 1-4 m, B-Spline fitting [1] SR3100, 1-5m, IT 2ms - 32ms,
6 degrees polynomial fitting [2]
Err
or(
cm
)
Distance(m)
10
0
1 4
20
15
25
5
-5
-10 1.5 2 2.5 3 3.5 4.5
![Page 24: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/24.jpg)
Due to the variation of the number of collected electrons during the integration time the repeatability of each depth point varies
Integration time-related Error
Integration Time: 30(ms) Integration Time: 80(ms)
30ms 80ms
![Page 25: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/25.jpg)
Light attenuates according to the law of inverse square
Amplitude Correction
Distance-based intensity correction [18]
![Page 26: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/26.jpg)
Large Sensor Pixel
In order to increase sensitivity,
- large pixel size or pixel binning
IR signal #2
)()(
)()(arctan
2)(
2211
4433
qqqq
qqqqcdt
IR signal #1
![Page 27: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/27.jpg)
Multiple light reflections between the lens and the sensor
Use scattering model [4] or anti-reflection material on lens
Light Scattering
Light scattering [4]
![Page 28: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/28.jpg)
Non-Local Means Filter
uses weighted average of all pixels using the similarity of squared neighborhoods
Depth Noise Reduction
Non-local denoising result [7]
![Page 29: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/29.jpg)
Bilateral Filter
uses weighted average of depth values from nearby pixels
Depth Noise Reduction
Bilateral filter kernel [9]
Bilateral filtering result
location Intensity
![Page 30: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/30.jpg)
Joint Bilateral Filter-based Method
refines depth values using color similarity [15]
Depth Super-resolution
(a) Color image 640x640 (b) Input depth map 64x64 (c) Refined depth maps 640x640 [15]
![Page 31: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/31.jpg)
ToF Motion Blur
Moving camera/object cause wrong depth calculation Motion blur
Image sensor
Moving Object Moving Object
![Page 32: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/32.jpg)
ToF Motion Blur
The characteristic of Tof motion blur is different from color
motion blur
Overshoot Blur
Undershoot Blur
Overshoot Blur
![Page 33: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/33.jpg)
ToF Motion Blur
The characteristic of Tof motion blur is different from color
motion blur
Reflected IR
4-Phase signals inside ToF camera
TimeInteg.
1Q
2Q
3Q
4Q
Radiated IR
))1(())1((
))1(())1((arctan
2)(
2211
4433
qnnqqnnq
qnnqqnnqcdt
![Page 34: RGBD Image Processing for 3D Modeling and Texturingcvlab.khu.ac.kr/icip_tutorial.pdf · 2015-03-23 · RGBD Image Processing for 3D Modeling and Texturing ICIP 2013 Tutorial Dr. Hwasup](https://reader033.fdocuments.us/reader033/viewer/2022042105/5e82e22586b8c4680d3fd408/html5/thumbnails/34.jpg)
ToF Motion Blur
Blur Detection
Blur Level
Input Depth
Deblurred Depth
- There are some relations
between Q1~Q4
1Q
2Q
3Q
4Q
4321 QQQQ
KQQQQ 4321
- We assume,
: single flat foreground
: single flat background