2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High...
-
Upload
joan-hines -
Category
Documents
-
view
215 -
download
0
Transcript of 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High...
![Page 1: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/1.jpg)
2012/11/11 1
High Quality Novel View Synthesis Based on Low Resolution Depth Image
and High Resolution Color Image
Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie
National Chung Cheng University
Taiwan
![Page 2: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/2.jpg)
2012/11/11 2
Outline Introduction Our hybrid camera system The proposed depth processing algorithm Experimental results Conclusion
![Page 3: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/3.jpg)
2012/11/11 3
Color + Depth in 3DTV/FTV Depth image, in combination with DIBR (Depth
Image-Based Rendering) technique, is important in multi-view 3DTV/FTV applications.
DIBR
view8 view1view2view3view4view5view6view7view9
Color image Depth image
![Page 4: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/4.jpg)
2012/11/11 4
Depth acquisition From stereo matching via disparity estimation
Unreliable for textureless regions Computationally intensive
From active photo-electrical sensing Lower resolution Higher acquisition speed and accuracy
Kinect SR4000
ToF
![Page 5: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/5.jpg)
2012/11/11 5
Hybrid camera systems High-resolution color cameras + Low-resolution depth
camera (mis-alignment between their optical axis) [12]: Use warped data of depth camera to refine ROI
disparity of the dual-eye camera Proposed: one color image is used to refine the warped (to
both sides) and up-sampled depth images
SR4000
Camera L Camera R
![Page 6: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/6.jpg)
2012/11/11 6
Processing diagram of our system
left color image
central depth image
right color image
Left preliminary
high-resolution depth
image processing
Right preliminary
high-resolution depth
image processing
Left high-quality
high-resolution depth
image processing
Right high-quality
high-resolution depth
image processing
Left depth image
Right depth image
![Page 7: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/7.jpg)
2012/11/11 7
Generation of preliminary high-resolution depth image Purpose: to generate high-resolution depth image
at the left and right sides
Depth image from
depth camera
Post-processing warped
depths to remove incorrect
depth values
Preliminary high
resolution depth images
Pre-processing depths to
remove noise
Warping and up-sampling
depth image to the
reference views
Calibrated camera parameters
![Page 8: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/8.jpg)
2012/11/11 8
Pre-processing of the depth map 33 median filter
Low resolution to high resolution depth warping Forward warping with a block size of 55 pixels
Post-processing by morphological operation Morphological opening
![Page 9: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/9.jpg)
2012/11/11 9
Generation of high quality high-resolution depth image Color-image-guided depth map refinement
No
1
Segment n and Db(x,y)
depth value is correct?
depth value is correct?
0
depth value D(x,y) with Db(x,y)=1 inside the segment n is averaged and used as the updated depth value
depth value D(x,y) with Db(x,y)=0 inside the segment n is averaged and used as the updated depth value
No
Yes
Yes
depth value is unchanged
Sn depth value is unchanged
![Page 10: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/10.jpg)
2012/11/11 10
Binarizing the warped depth image to have foreground/background classification
Color segmentation by mean-shift algorithm Label each segment as FG or BG
Refine the corresponding depth values according to the segment categorization
Bilateral filtering is performed for the edge parts of the modified depth image
![Page 11: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/11.jpg)
2012/11/11 11
Experiments – system configurations Color image: 640x480 Depth image: 176x144 (SR4000) Intel Core 2 Duo Q6600 2.33 GHz, and DDR2
800 4GB
![Page 12: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/12.jpg)
2012/11/11 12
Preliminary high-resolution depth image
(a) (c) (e)
(b) (d) (f)
After depth warping
After morphological operation
Traditional depth warping
![Page 13: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/13.jpg)
2012/11/11 13
High-quality high-resolution depth image
(a) (c) (e)
(b) (d) (f)
Mean-shift
Incorrect pixels
After correction
Bilateral filtering
![Page 14: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/14.jpg)
2012/11/11 14
A series of generated depth maps
![Page 15: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/15.jpg)
2012/11/11 15
Virtual view synthesis based on various depth maps
(a) (c) (e)
(b) (d) (f)
Bilateral filtering
Bilateral filtering on edges
Bilateral filtering on edges + Gaussian
Hole filled
![Page 16: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/16.jpg)
2012/11/11 16
Speed performance
ProcedureTime (sec)
Pre-processingDepth Warping 0.117
Color segmentation 0.742
Post-processingBilateral filtering on edges 0.352
Gaussian filtering 0.104
Novel view synthesis by DIBR (including hole filling) 0.144
![Page 17: 2012/11/111 High Quality Novel View Synthesis Based on Low Resolution Depth Image and High Resolution Color Image Jui-Chiu Chiang, Zheng-Feng Liu, and.](https://reader036.fdocuments.us/reader036/viewer/2022062717/56649e5d5503460f94b56548/html5/thumbnails/17.jpg)
2012/11/11 17
Conclusion A hybrid camera system consisting of high-
resolution color camera and low-resolution depth camera is proposed
A high-quality high-resolution depth images can be obtained for the left and right eyes – multi-view video plus depth (N=2 channels).
Future work Combine Kinect device and stereo camera to get better
depth image and cheaper cost