Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese...
Transcript of Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese...
![Page 1: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/1.jpg)
Xue Tianfan, Liu Jianzhuang, Tang Xiaoou Multimedia Lab
The Chinese Univ. of Hong Kong
![Page 2: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/2.jpg)
ObjectiveRecovering 3D geometry from single image
Reflection‐symmetric &Piecewise Planner
![Page 3: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/3.jpg)
Related worksPrevious approaches
Focus on a specific kind of objectFace [V. Blanz, SIGGRAPH 99]
Car [M. J. Leotta., CVPR 09]
Add user interaction[L. Zhang, CVPR 01]
Our approach
Automatically recover 3D geometry from a single view of symmetric object
![Page 4: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/4.jpg)
Our approach: Recovering 3D Geometry using symmetry
Symmetry is ubiquitous
Symmetry provides rich information for reconstruction
View difference in stereo View difference in symmetry
![Page 5: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/5.jpg)
Reconstruction – Pipeline
Symmetric &Piecewise Planner
![Page 6: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/6.jpg)
Reconstruction – Plane DetectionPlane Detection
A symmetric line pair can determine a plane
Some heuristic rules are proposed to get a set possible symmetric line pair
1π1l2l
3l4l2π 4π3π
1l2l
3l4l
![Page 7: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/7.jpg)
Reconstruction – Plane DetectionPlane Detection
How to determine whether two lines are symmetric
Suppose all the pairs are symmetric
l1l2 l3
l4
l1l2 l3 l4
l1
l2
l3 l4
11l
12l
31l
32l 3
3l 34l
Camera Center
(a)
(b) (c)
1π
2π
3π
24l
23l
Image Plane
SymmetryPlaneπ
13l 1
4l
![Page 8: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/8.jpg)
Reconstruction – Plane DetectionPlane Detection
(li,lj,πk): πk is the plane determined by li,ljP: The set of all line pairsP’: The set of correct line pairs
plane(P’): The set of planes determined by the pairs in P’
![Page 9: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/9.jpg)
Reconstruction – Plane LabelingPlane Labeling
Assign each pixel to the plane it belongs to
This problem is modeled as a Markov Random Field
1π
2π 4π
3π
![Page 10: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/10.jpg)
Reconstruction – Plane LabelingPlane
Energy function:
Data Term
pππ
![Page 11: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/11.jpg)
Reconstruction – Plane LabelingPlane labeling
Smooth Term
( , )p qπ πΔpπ
qπsymπ
pπ
qπ
Symmetry Term
![Page 12: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/12.jpg)
Experiment Result
J1
J2
J3
J4
J5
K1
K2
K3
K4
K5
L1
L2
L3
L4
L5
M1
M2
M3
M4
M5
N1
N2
N3
N4
N5
O1
O2
O3
O5
P1
P2
P3
P4
P5
Q1
Q2
Q3
Q4
Q5
O4
![Page 13: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/13.jpg)
Conclusion
ConclusionSymmetry is a powerful tool for recovering 3D geometry
We focus on reflection symmetry
Another works in this year CVPR on translation symmetryC. Wu, J. Frahm, M. Pollefeys “Repetition‐based Dense Single‐View
Reconstruction”, CVPR, 2011.
Possible ExtensionsSemi‐automatic 3D reconstruction using symmetry
Combining 3D reconstruction with 3D retrieval
Improve the learning based 3D reconstruction using symmetry
![Page 14: Xue Liu Jianzhuang, Tang Xiaoou Multimedia Lab The Chinese ...vision.cse.psu.edu/research/symmComp/workshop/HK... · Multimedia Lab The Chinese Univ. of Hong Kong. Objective yRecovering](https://reader033.fdocuments.us/reader033/viewer/2022052018/603103f8f6557c6c89192b73/html5/thumbnails/14.jpg)
Q&A