Layered Segmentation and Optical Flow Estimation Over...
1
Layered Segmentation and Optical Flow Estimation Over Time Deqing Sun 1 , Erik B. Sudderth 1 , and Michael J. Black 1,2 1 Department of Computer Science, Brown University, Providence, RI, USA, 2 Max Planck Institute for Intelligent Systems, 72076 Tubingen, Germany Acknowledgements: DS and MJB were supported in part by the NSF Collaborative Research in Computational Neuroscience Program (IIS-0904875) and a gift from Intel Corporation. +2 −1 +1 1 2 1 , 1 2 , 2 3 , 3 ∗ Layer 1 support Layer 2 support Segmentation Composite flow Layer 1 flow Layer 2 flow Layer 3 (background) flow MIT Layer Segmentation Benchmark Ground truth Flow field nLayers Layers++ [5] Layer segmentation Flow field Classic+NL [4] Toy Example Multiple Frames Help Error Flow field Layer segmentation End-point error: 0.345 End-point error: 0.311 t+2 t+1 t t-1 Using frames t and t+1 Using all 4 frames Darker: larger Middlebury Optical Flow Benchmark Public table (June 2012) Classic+NL [4] nLayers MDP-Flow2 [8] Ground truth Black: occlusion Layer segmentation [1] Grundmann, Kwatra, Han, & Essa. Efficient hierarchical graph-based video segmentation. CVPR, 2010. [2] Liu, Freeman, Adelson, & Weiss. Human-assisted Motion Annotation. CVPR, 2008. [3] Sudderth & Jordan, Shared Segmentation of Natural Scenes Using Dependent Pitman-Yor Processes. NIPS, 2009. [4] Sun, Roth, & Black, Secrets of Optical Flow Estimation and Their Principles. CVPR, 2010. [5] Sun, Sudderth, & Black, Layered Image Motion with Explicit Occlusions, Temporal Consistency and Depth Ordering. NIPS, 2010. [6] Wang & Adelson, Representing Moving Images with Layers. IEEE TIP 1994. [7] Weiss, Smoothness in Layers: Motion Segmentation Using Nonparametric Mixture Estimation. CVPR, 1997. [8] Xu, Jia, & Matsushita. Motion Detail Preserving Optical Flow Estimation. IEEE TPAMI to appear. References Occluded Occlusion Reasoning Robust penalty (generalized Charbonnier) Constant penalty (uniform distribution) Visible Discrete-Continuous Optimization After discrete optimization After continuous Optimization [5] After affine K-means clustering Optical flow [4] +1 +1∗ ∗ +1, −1 Motion Support function Segmentation Extended Alpha Expansion Segmentation move simultaneously changes several support functions to change segmentation 1− 1 proposal 1 old 2 proposal 2 old 1 2 Near Far HGVS [1] Rand Index (RI): 0.766 RI: 0.689 RI: 0.499 nLayers RI: 0.979 RI: 0.766 RI: 0.881 Generative Layered Model More complex moves can simultaneously change segmentation and flow. The flow field is chosen from a candidate set of flow fields, which are adaptively refined by continuous optimization. Flow field Layer segmentation Color key
Transcript of Layered Segmentation and Optical Flow Estimation Over...
Layered Segmentation and Optical Flow Estimation Over Time
Deqing Sun1, Erik B. Sudderth1, and Michael J. Black1,2
1Department of Computer Science, Brown University, Providence, RI, USA, 2Max Planck Institute for Intelligent Systems, 72076 Tubingen, Germany
Acknowledgements: DS and MJB were supported in part by the NSF Collaborative Research in Computational Neuroscience Program (IIS-0904875) and a gift from Intel Corporation.
𝑰𝑡+2 𝑰𝑡−1 𝑰𝑡+1 𝑰𝑡
𝒈𝑡1
𝒈𝑡2
𝒖𝑡1, 𝒗𝑡1
𝒖𝑡2, 𝒗𝑡2
𝒖𝑡3, 𝒗𝑡3 𝒌𝑡∗
Layer 1
support
Layer 2
support
Segmentation
Composite flow
Layer 1 flow
Layer 2 flow
Layer 3
(background) flow
MIT Layer Segmentation Benchmark
Ground truth Flow field
nLayers
Layers++ [5]
Layer segmentation Flow field
Classic+NL [4]
Toy Example Multiple Frames Help
Error Flow field Layer segmentation
End-point error: 0.345
End-point error: 0.311
t+2 t+1 t t-1
Using
frames t
and t+1
Using all 4
frames
Darker: larger
Middlebury Optical Flow Benchmark
Public table (June 2012)
Classic+NL [4] nLayers MDP-Flow2 [8] Ground truth
Black: occlusion
Layer segmentation
[1] Grundmann, Kwatra, Han, & Essa. Efficient hierarchical graph-based video segmentation. CVPR, 2010.
[2] Liu, Freeman, Adelson, & Weiss. Human-assisted Motion Annotation. CVPR, 2008.
[3] Sudderth & Jordan, Shared Segmentation of Natural Scenes Using Dependent Pitman-Yor Processes. NIPS, 2009.
[4] Sun, Roth, & Black, Secrets of Optical Flow Estimation and Their Principles. CVPR, 2010.
[5] Sun, Sudderth, & Black, Layered Image Motion with Explicit Occlusions, Temporal Consistency and Depth Ordering. NIPS,
2010.
[6] Wang & Adelson, Representing Moving Images with Layers. IEEE TIP 1994.
[7] Weiss, Smoothness in Layers: Motion Segmentation Using Nonparametric Mixture Estimation. CVPR, 1997.
[8] Xu, Jia, & Matsushita. Motion Detail Preserving Optical Flow Estimation. IEEE TPAMI to appear.
References
Occluded
Occlusion Reasoning
Robust penalty (generalized Charbonnier)
Constant penalty (uniform distribution)
Visible
Discrete-Continuous Optimization
After discrete
optimization
After continuous
Optimization [5]
After affine K-means
clustering
Optical flow [4]
𝑰𝑡 𝑰𝑡+1
𝒖𝑡𝑘
𝒗𝑡𝑘
𝒌𝑡+1∗ 𝒌𝑡∗
𝐾
𝐾 𝐾
𝒈𝑡+1,𝑘 𝒈𝑡𝑘
𝐾 − 1
Motion
Support function
Segmentation
Extended Alpha Expansion Segmentation move simultaneously changes several support functions to change segmentation
𝐛𝑡 1 − 𝐛𝑡
𝐠𝑡1proposal
𝐠𝑡1old
𝐠𝑡2proposal
𝐠𝑡2old
𝐠𝑡1
𝐠𝑡2
Near
Far
HGVS [1]
Rand Index (RI): 0.766
RI: 0.689
RI: 0.499
nLayers
RI: 0.979
RI: 0.766
RI: 0.881
Generative Layered Model
More complex moves can simultaneously change segmentation and flow. The flow field is chosen from a
candidate set of flow fields, which are adaptively refined by continuous optimization.
Flow field Layer
segmentation
Color key
![Robust Statistical Estimation and Segmentation of …cs294-6/fa06/papers/RANSAC25-Yang-A… · Robust Statistical Estimation and Segmentation of Multiple Subspaces ... [5,16,34].](https://static.fdocuments.us/doc/165x107/5ad8473b7f8b9a991b8d1c81/robust-statistical-estimation-and-segmentation-of-cs294-6fa06papersransac25-yang-arobust.jpg)
