MoSculp: Interactive Visualization of Shape and...
Transcript of MoSculp: Interactive Visualization of Shape and...
MoSculp: Interactive Visualization of Shape and Time
XiumingZhang1
William T. Freeman1,2
TaliDekel1,2
TianfanXue1,2
Stefanie Mueller1
QiuruiHe1,2
Andrew Owens1,3
JiajunWu1
1 MIT CSAIL 2 Google Research 3 UC Berkeley
1
2
3
4
5
6Video Courtesy of Tom Buehler (MIT CSAIL)
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
7
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
8
Motivation
9
Muybridge, The Human Figure in Motion, 1901
Motivation
10
Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive, 1954
Motivation
11
Muybridge, The Human Figure in Motion, 1901 Edgerton, Back Dive, 1954Duchamp, Nude
Descending a Staircase, No. 2, 1912
Related Work
12
Edgerton, Stroboscopic Photography, 1927–1931
2D
Related Work
13
Edgerton, Stroboscopic Photography, 1927–1931
2D
Freeman & Zhang, Shape-Time Photography, CVPR ’03
Requires a depth camera
Related Work vs. Ours
14
Edgerton, Stroboscopic Photography, 1927–1931
2D
Freeman & Zhang, Shape-Time Photography, CVPR ’03
Requires a depth camera
MoSculp
3D w/ an RGB camera
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
15
System Walkthrough
16
17
18
19
20
21
22
23
24
25
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
26
User Studies: Design Choices
27
With Floor Reflections WithoutPreferred by 82%
User Studies: Efficacy in Conveying Motion
28
Baseline 1 (Stroboscopic)
Baseline 2(Shape-Time)
MoSculpPreferred by 75%
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
29
30
Input Video Motion Sculpture Generation
Depth-Preserving Compositing
3D Shape & Pose Estimation
… …
Overview
31
Input Video Motion Sculpture Generation
Depth-Preserving Compositing
3D Shape & Pose Estimation
… …
Overview
Approach: 2D Keypoint Detection
32
Time: t
[Cao et al., CVPR ’17]
Approach: 2D Keypoint Detection
33
[Cao et al., CVPR ’17]
Time: t + 1
34
Hidden Markov Model
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
35
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
• Small reprojection error
362D Image3D Model
[Loper et al., ToG ’15]
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
• Small reprojection error • Large probability of the poses
37
p( ) > p( )
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
• Small reprojection error • Large probability of the poses• Smooth evolution of poses
38
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
• Small reprojection error • Large probability of the poses• Smooth evolution of poses
39
Per-Frame Optimization[Bogo et al., ECCV ’16]
Original Camera View Novel View
Approach: 3D Estimation• Solve for the best shape and poses jointly for the clip
• Small reprojection error • Large probability of the poses• Smooth evolution of poses
40
Original Camera View Novel View
Our Joint Optimization
41
Input Video Motion Sculpture Generation
Depth-Preserving Compositing
3D Shape & Pose Estimation
… …
Overview
Approach: Sculpture Generation
42
43
44
45
Input Video Motion Sculpture Generation
Depth-Preserving Compositing
3D Shape & Pose Estimation
… …
Overview
Approach: Compositing• Key challenge: how to “put together” 3D sculpture and 2D video?
46
Approach: Compositing• Naive Compositing: sculpture on top of the frames
47
Approach: Compositing• Full 3D Rendering: texturing the 3D models
48
Skirt Not Covered by 3D Model
Approach: Compositing• Solution: depth-preserving composite
49
Rendered Depth (Refined)
Approach: Compositing• Solution: depth-preserving composite
50
Approach: Before Refinement
51
Approach: After Refinement
52
Outline• Related Work• System Walkthrough• User Studies• Approach• Results• Conclusion
53
54
55
56
Single-Frame Shape and Pose Estimation
57
Our Joint Estimation
58
59
60
61
62
Texture from
Original Frames
63
64
65
Handling a Moving Camera
66
Outline• Related Work• System Walkthrough• User Studies• Approach• Results • Conclusion
67
Limitation: Repeated, Localized Motion
68
69
Conclusion
70
Video Courtesy of Tom Buehler (MIT CSAIL)
Please come to our demo D-12 for more!
Thank you!71
http://mosculp.csail.mit.edu