Automatic Portrait Segmentation and...
Transcript of Automatic Portrait Segmentation and...
![Page 1: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/1.jpg)
Automatic Portrait Segmentation and Matting
Xiaoyong ShenThe Chinese University of Hong Kong
![Page 2: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/2.jpg)
Research on CV
• Pixel based (low level/ early vision)• Filtering, restoration, denoise, enhancement, deblur,
editing, dehaze, etc.
• Region/ Patch based (Middle level vision)• Matching, optical flow, stereo matching, tracking,
segmentation, etc.
• Object/ Semantic based (high level vision)• Semantic segmentation, Object detection, image
classification, recognition, etc.
![Page 3: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/3.jpg)
My Research on CV
• Pixel based (low level vision)• Filtering, restoration, denoise, enhancement, deblur,
editing, dehaze, etc.
• Region/ Patch based (Middle level vision)• Matching, optical flow, stereo matching, tracking,
segmentation, etc.
• Object based (high level vision)• Semantic segmentation, Object detection, image
classification, recognition, etc.
![Page 4: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/4.jpg)
![Page 5: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/5.jpg)
Multi-Spectral Image Restoration
• Input• Noisy RGB image I0
• E.g. captured at night
• Clean guidance image G• E.g. dark-flashed NIR, or flashed RGB images
• Output• Denoised image I
• Structures are clear as guidance G.• Appearance is the same as image I0.• Shadow/Highlight does not affect.
5[TPAMI 2015]
![Page 6: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/6.jpg)
Scale Map
• Given 𝐼∗ – the expected ground truth noise-free image, our scale map s is defined under the following condition
min 𝛻𝐼∗ − 𝑠𝛻𝐺
• It adapts structures of 𝐺 to that of I*.
• It is an ideal ratio map between 𝛻𝐺 and 𝛻𝐼∗.
6
![Page 7: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/7.jpg)
Result
7Our Result Ground Truth
Input Noisy Image Input NIR Image
![Page 8: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/8.jpg)
RGB Input I
8
![Page 9: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/9.jpg)
NIR Input G
9
![Page 10: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/10.jpg)
BM3D
10
![Page 11: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/11.jpg)
Our Result
11
![Page 12: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/12.jpg)
Mutual-Structure Filter
[ICCV 2015 Oral Presentation]
![Page 13: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/13.jpg)
Depth/RGB Restoration
Noisy Depth
![Page 14: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/14.jpg)
Depth/RGB Restoration
Noisy RGB Image
![Page 15: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/15.jpg)
Depth/RGB Restoration
Ground truth
![Page 16: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/16.jpg)
Depth/RGB Restoration
OursPSNR = 37.19
![Page 17: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/17.jpg)
Rolling Guidance Filter
One line code only: 𝐼𝑡+1 = 𝐽𝐹(𝐼0, 𝐼𝑡)
[ECCV 2014 Oral Presentation]
![Page 18: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/18.jpg)
Texture Removal
18
![Page 19: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/19.jpg)
Halftone Image
19
![Page 20: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/20.jpg)
De-Filter
One line code only: 𝐼𝑡+1 = 𝐼𝑡 + (𝐼0 − 𝐹(𝐼𝑡))
![Page 21: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/21.jpg)
Reverse Skin Retouch
Retouched input
![Page 22: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/22.jpg)
Reverse Skin Retouch
Reversed
![Page 23: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/23.jpg)
Reverse Skin Retouch
Before retouch
![Page 24: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/24.jpg)
![Page 25: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/25.jpg)
Multi-Spectral Matching
• Match general multi-spectral images with significant displacement and obvious structure inconsistency
Different Exposures RGB/Depth RGB/NIR Flash/No-flash
![Page 26: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/26.jpg)
Result
• Match RGB/NIR image pair
InputsOur ResultBlended
![Page 27: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/27.jpg)
Applications
• HDR construction
Without AlignmentWith AlignmentConstructed HDR
![Page 28: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/28.jpg)
Internet Image Matching
Reference Input
Dense Correspondences ?
Exist Correspondence
No Correspondence
[SIGGRAPH ASIA 2016]
![Page 29: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/29.jpg)
Our Motivation
Reference Input
Dense Correspondences ?
Foremost Region Matching
![Page 30: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/30.jpg)
Time-lapse Generation
![Page 31: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/31.jpg)
Automatic Morphing
![Page 32: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/32.jpg)
Automatic Morphing
![Page 33: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/33.jpg)
Object-based MatchingAchieve higher accuracy with the help of object (person)
![Page 34: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/34.jpg)
Object-based Matching
State-of-the-art Ours
![Page 35: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/35.jpg)
![Page 36: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/36.jpg)
Classification and Segmentation
• Fine-grained Classification• DeepLAC (CVPR 2015)
• Text detection and recognition
• Semantic object segmentation• Portrait segmentation and matting
• VOC challenge
![Page 37: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/37.jpg)
Automatic Portrait Segmentation
![Page 38: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/38.jpg)
Motivation
• Abundant portraits in smartphone photos
38
Portrait, 30%
Others, 70%
Samsung UK
Portrait, 90%
Others, 10%
Symon Whitehorn from HTC
![Page 39: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/39.jpg)
Portrait Post-processing
39
![Page 40: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/40.jpg)
Foreground Selection
40
![Page 41: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/41.jpg)
Quick Selection
41
![Page 42: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/42.jpg)
Automatic Segmentation
42
Automatic?
![Page 43: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/43.jpg)
Challenges
43
Similar Color Complex Background Various Accessories
Low Contrast Diverse PoseComplicated Edges
![Page 44: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/44.jpg)
Possible Solutions
• Graph-cut with face tracker
44
![Page 45: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/45.jpg)
Possible Solutions
• CNNs for semantic segmentation
45
![Page 46: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/46.jpg)
Most Related Work
• Interactive Image Selection• Lazy snapping [Li et al. 2004]• Grabcut [Rother et al. 2004]• Paint Selection [Li et al. 2009]
• CNNs for Semantic Object Segmentation• FCN [Long et al. 2014]• DeepLab [Chen et al. 2014]• CRFasRNN [Zheng et al. 2015]
• Image Matting• Bayesian matting [Chuang et al. 2001]• Closed-form matting [Levin et al. 2008]• KNN matting [Chen et al. 2013]
46
![Page 47: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/47.jpg)
Our Approach
47
PortraitFCN and PortraitFCN+
![Page 48: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/48.jpg)
Our System
48
Detector
Conv ReLUPooling Conv
ConvPoolingReLU
DeConv Mask[Long et al. 2015]
PortraitFCN ModelRGB Channels 2 Outputs
![Page 49: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/49.jpg)
PortraitFCN
49
• Fine tune it from original FCN-8s model
Portrait Knowledge
![Page 50: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/50.jpg)
PortraitFCN+
50
Detector
Conv ReLUPooling Conv
ConvPoolingReLU
DeConv Mask[Long et al. 2015]
PortraitFCN+ ModelRGB+Shape+Position 2 Outputs
Shape Position
![Page 51: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/51.jpg)
Shape Channel
51
……
Labeled Masks
Align
Canonical Pose
Mean
Shape Channel
𝑀 =σ𝑖𝑤𝑖 ∘ 𝑇𝑖(𝑀𝑖)
σ𝑖𝑤𝑖
Align
Test Image
![Page 52: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/52.jpg)
Position Channel
52
Canonical Pose
x- Coordinate y- Coordinate
Position Test Image
Align
![Page 53: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/53.jpg)
Effectiveness
53
Input
![Page 54: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/54.jpg)
Effectiveness
54
PortraitFCN
![Page 55: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/55.jpg)
Effectiveness
55
PortraitFCN+
![Page 56: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/56.jpg)
Experiments and Applications
56
![Page 57: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/57.jpg)
Our Dataset
• 1,800 portraits from Flickr with labeled mask• 1500 portraits as the training data
• 300 for testing
• Large variations on portrait types• Age, color, background, clothing, accessories, head
position, hair style, lighting, etc.
57
![Page 58: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/58.jpg)
58
![Page 59: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/59.jpg)
Training
• Fine turn the model starting from FCN-8s• Synthesize more data with different transforms
• Using the person class and background weights
• Find the best learning rate• Loss
• accuracy
59
![Page 60: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/60.jpg)
Find the Best LR
60
![Page 61: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/61.jpg)
Evaluation
61
Methods Mean IoU (%)
Graph-cut 80.02
FCN (Person Class) 73.09
IoU =area(output ∩ ground truth)
area(output ∪ ground truth)
![Page 62: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/62.jpg)
Evaluation
62
Methods Mean IoU (%)
Graph-cut 80.02
FCN (Person Class) 73.09
PortraitFCN 94.20
IoU =area(output ∩ ground truth)
area(output ∪ ground truth)
![Page 63: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/63.jpg)
Evaluation
63
Methods Mean IoU (%)
Graph-cut 80.02
FCN (Person Class) 73.09
PortraitFCN 94.20
PortraitFCN+ (Only with Mean Mask) 94.89
PortraitFCN+ (Only with Normalized x and y) 94.61
IoU =area(output ∩ ground truth)
area(output ∪ ground truth)
![Page 64: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/64.jpg)
Evaluation
64
Methods Mean IoU (%)
Graph-cut 80.02
FCN (Person Class) 73.09
PortraitFCN 94.20
PortraitFCN+ (Only with Mean Mask) 94.89
PortraitFCN+ (Only with Normalized x and y) 94.61
PortraitFCN+ 95.91
IoU =area(output ∩ ground truth)
area(output ∪ ground truth)
![Page 65: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/65.jpg)
Comparisons
65
Input
![Page 66: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/66.jpg)
Comparisons
66
Ground Truth
![Page 67: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/67.jpg)
Comparisons
67
Graph-cut
![Page 68: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/68.jpg)
Comparisons
68
FCN-8s (Person)
![Page 69: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/69.jpg)
Comparisons
69
PortraitFCN
![Page 70: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/70.jpg)
Comparisons
70
PortraitFCN+
![Page 71: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/71.jpg)
Comparisons
71
Input Ground Truth
IoU = 0.83 IoU = 0.42
IoU = 0.91 IoU = 0.85
FCN-8s Graph-cut
IoU = 0.99
IoU = 0.98
Ours
![Page 72: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/72.jpg)
Comparisons
72
Input Ground Truth
IoU = 0.77 IoU = 0.95
IoU = 0.38 IoU = 0.84
FCN-8s Graph-cut
IoU = 0.98
IoU = 0.98
Ours
![Page 73: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/73.jpg)
Comparisons
73
Input Ground Truth
IoU = 0.83 IoU = 0.53
IoU = 0.81 IoU = 0.89
FCN-8s Graph-cut
IoU = 0.99
IoU = 0.98
Ours
![Page 74: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/74.jpg)
Robustness
74
Color Scale Rotation Occlusion
![Page 75: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/75.jpg)
User Study
• Our result provides very good initialization for further refinement
75
![Page 76: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/76.jpg)
Segmentation is not enough--Automatic Portrait Matting
![Page 77: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/77.jpg)
Portrait Matting
Input Image Alpha Matte
Color transform Depth-of-field Portrait
Stylization Cartoon
Background Edit
![Page 78: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/78.jpg)
Problem Definition
78
𝜶𝑭 + 𝟏 − 𝜶 𝑩
foreground background
Image Alpha/foreground opacity
𝑰 =
![Page 79: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/79.jpg)
Natural Image Matting
• Color Sampling Methods• Given manual-labeled trimap
• Bayesian Matting [Y-Y Chuang, 2001], etc.
79
Image Trimap Alpha matte
![Page 80: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/80.jpg)
Natural Image Matting
• Propagation approaches• Given manual-labeled strokes & trimap
• Closed-form Matting [Levin, 2008], etc.
80
𝛼 = 𝑎𝑟𝑔𝑚𝑖𝑛 𝛼𝑇𝐿𝛼 + 𝜆 𝛼 − 𝑏𝑠𝑇𝐷(𝛼 − 𝑏𝑠)
Matting Laplacian User-provided Strokes
Diagonal stroke mask
![Page 81: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/81.jpg)
Motivation
• It is very hard to specify trimap or strokes
81
Input Labeled Strokes Closed-form Matting
error
![Page 82: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/82.jpg)
Motivation
• It is very hard to specify trimap or strokes
Input Labeled Trimap Closed-form Matting
error
![Page 83: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/83.jpg)
Motivation
83
Usually we need to refine the trimap many times to get a good alpha matte……
![Page 84: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/84.jpg)
Segmentation to Matting
![Page 85: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/85.jpg)
Segmentation to Matting
![Page 86: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/86.jpg)
86
![Page 87: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/87.jpg)
Learning for Automatic Matting
• Challenges• Data preparation
• Learning framework
• We propose end-to-end Convolutional Neural Networks (CNNs) for Portrait Matting
87
![Page 88: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/88.jpg)
Learning Data Collection
• 2000 portraits from Flickr with large variation• Keywords…
• Different Age, gender, pose, hairstyle, background…
• Different camera type…
• Data example
88
![Page 89: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/89.jpg)
8989
![Page 90: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/90.jpg)
Data Labeling
• Apply closed-form matting and robust matting• Gradually refine the input trimap
• Choose the best one from closed-form or robust matting
• User interface
• Ground truth example
90
![Page 91: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/91.jpg)
9191
![Page 92: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/92.jpg)
Learn Automatic Matting
92
![Page 93: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/93.jpg)
Our Method
93
Trimap labeling• Input: RGB image
• Output: trimap
• Network: Fine tuned from FCN
![Page 94: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/94.jpg)
Our Method
94
Image Matting Layer• Input: trimap
• Output: alpha matte
• Novel-designed structure
![Page 95: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/95.jpg)
Our Method
95
Image Matting Layer• Feed-Forward:
𝑚𝑖𝑛 𝜆𝐴𝑇𝐵𝐴 + 𝜆 𝐴 − 1 𝑇𝐹(𝐴 − 1) + 𝐴𝑇𝐿𝐴• Back-Forward:
𝜕𝑓
𝜕𝐵= −𝜆𝐷−1𝑑𝑖𝑎𝑔(𝐷−1𝐹)
𝜕𝑓
𝜕𝐹=𝜕𝑓
𝜕𝐵+ 𝐷−1
𝜕𝑓
𝜕𝜆= −𝜆𝐷−1𝑑𝑖𝑎𝑔 𝐹 + 𝐵 𝐷−1𝐹
![Page 96: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/96.jpg)
Our Method
96
Image Matting Layer• Loss function:
𝐿(𝐴, 𝐴𝑔𝑡) =
𝑖
𝑤 𝐴𝑖𝑔𝑡
| 𝐴𝑖 − 𝐴𝑖𝑔𝑡
|,
𝑤 𝐴𝑖𝑔𝑡
= −𝑙𝑜𝑔(𝑝(𝐴 = 𝐴𝑖𝑔𝑡))
![Page 97: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/97.jpg)
Model Training
97
• Data augmentation• 4 scales {0.6,0.8,1.2,1.5}
• 4 rotations {-45,-22,22,45} degree
• Gamma value {0.5,0.8,1.2,1.5}
• Network initialization• Fine tuned from FCN-8s Model [J. Long, 2015]
![Page 98: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/98.jpg)
Experiments
98
• Running Time• Training time: 20k iterations, one day on Titan X GPU
• Testing Time: 0.6s for 600×800 color image.
• Comparisons• Graph-cut
• FCN Baseline: direct FCN segmentation followed by closed-form matting
![Page 99: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/99.jpg)
Results
99
Input Graph-cut FCN Ours
![Page 100: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/100.jpg)
Results
100
Input Graph-cut FCN Ours
![Page 101: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/101.jpg)
Results
101
Input Graph-cut FCN Ours
![Page 102: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/102.jpg)
Results
102
Input Graph-cut FCN Ours
![Page 103: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/103.jpg)
Failure Cases
103
Input Alpha Matte Input Alpha Matte
![Page 104: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/104.jpg)
Applications
104
Input Stylization PS GS Stick PS Fresco Stylization
Input Stylization Depth-of-Field PS Fresco Stylization
![Page 105: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/105.jpg)
Applications
105
Input Stylization PS Palette Knife PS GS Stick PS Sketch
Input PS Oil Paint Depth-of-Field PS GS Stick Stylization
![Page 106: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/106.jpg)
Applications
106
Input Stylization PS Palette Knife Depth-of-Field Stylization
Input Stylization PS Palette Knife PS Dark Stroke PS Paint Daubs
![Page 107: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/107.jpg)
Conclusions
• High accuracy automatic portrait segmentation and matting approach• A novel CNN framework• Training and testing dataset• Benefits lots of applications
• Future work• Video segmentation• Human segmentation• Single portrait image depth estimation• Weakly supervised version
107
![Page 108: Automatic Portrait Segmentation and Mattingvalser.org/webinar/slide/slides/20160817/XiaoyongSheng.pdf · 17/08/2016 · [ICCV 2015 Oral Presentation] Depth/RGB Restoration Noisy](https://reader035.fdocuments.us/reader035/viewer/2022071000/5fbc88049944475ec10f6641/html5/thumbnails/108.jpg)
Q & A
108
Thanks