Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability...
Transcript of Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability...
![Page 1: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/1.jpg)
Probabilistic Graphical Models
Michael Yang
September 12, 2017
![Page 2: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/2.jpg)
• Since 2016, Assistant Professor, University of Twente
• 2008-2011, Ph.D, University of Bonn
• 2016-2020, Co-Chair ISPRS WG Dynamic Scene Analysis
• Main Research Areas: Photogrammetry, Computer Vision
Brief CV
2
![Page 3: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/3.jpg)
• Introduction
• Random Fields
Man-made Object Segmentation Semantic Video Segmentation
Outline
3
![Page 4: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/4.jpg)
Applications
4
• Medical diagnosis
• Social network models
• Speech recognition
• Robot localization
•Remote sensing
• Natural language processing
• Computer vision– Image segmentation– Tracking– Scene understanding– Image classification– 3D reconstruction
•........
6
![Page 5: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/5.jpg)
Applications
Segmentation
5
7Yang, Rosenhahn, 2016
![Page 6: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/6.jpg)
Applications
Classification
6
Zhong & Wang 2011
• Reading letters/numbers
• Land-cover classificationin remote sensing
8
![Page 7: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/7.jpg)
Applications
Interpretation
7
• Building and road extraction
• Facade interpretation
Chai et al., 2013
Yang & Förstner, 2011
9
![Page 8: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/8.jpg)
Probabilistic Graphical Models
are a marriage between
probability theory & graph theory
8
![Page 9: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/9.jpg)
Bayesian networks
Graphical Models
Conditional/Markov random fields
9
![Page 10: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/10.jpg)
• Graph
Graphical Models
set of the nodes
set of the undirected edges
set of the directed edges
10
![Page 11: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/11.jpg)
• Graphical models
A stochastical model represented by a graph
Graphical Models
• Nodes represent random variables
• Edges represent mutual relationships
Undirected edges model joint probabilities
Directed edges model conditional dependencies
11
![Page 12: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/12.jpg)
• Graphical models
Graphical Models
• Visualization of dependencies
• Conditional probabilities : directed edges(Bayesian Networks)
• Joint probabilities: undirected edges(Markov Random Field)
12
![Page 13: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/13.jpg)
• Introduction
• Random Fields
Man-made Object Segmentation Semantic Video Segmentation
Outline
13
![Page 14: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/14.jpg)
• Definition Markov random field : graphical model over an undirected graph+ positivity property + Markov property
Markov property:
MRFs
Set of random variables linked to nodes
Set of neighbored random variable
14
![Page 15: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/15.jpg)
• Pairwise MRFspopular
with energy function
MRFs
15
![Page 16: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/16.jpg)
• Structure of MRFsTypical graph structures
MRFs
rectangular grid irregular graph pyramid structure
Figure courtesy of P. Perez
16
![Page 17: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/17.jpg)
• Image Denoising using Pairwise MRFs
MRFs
[From Bishop PRML] noisy image result
17
![Page 18: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/18.jpg)
• Definition: conditioanl random fields
A CRF is an MRF globally conditioned on observed data
CRFs
18
![Page 19: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/19.jpg)
• Definition: conditioanl random fields
A CRF is an MRF globally conditioned on observed data
CRFs
Conditional distribution
Joint distributionMRF
CRF
19
![Page 20: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/20.jpg)
• Introduction
• Random Fields
Man-made Object Segmentation Semantic Video Segmentation
Outline
20
![Page 21: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/21.jpg)
CRFs
Yang & Förstner, 2011
Region adjacency graphBuilding facade image
21
![Page 22: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/22.jpg)
CRFs
CRF has a Gibbs distribution
Gibbs energy function (all dependent on data)
22
![Page 23: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/23.jpg)
Yang & Förstner, 2011
Region adjacency graph
Region hierarchy graphMulti-layer CRFBlue edges
Red edges
(a) Test image (b) Multi-scale segmentation
(c) Graphical model
Hierarchical CRFs
23
![Page 24: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/24.jpg)
Unary potential: classifier output (RF)Pairwise potential: (Data-dependent) PottsHierarchical potential: (Data-dependent) Potts
Energy function
Hierarchical CRFs
Michael Yang 24
![Page 25: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/25.jpg)
25
Scene Interpretation
Workflow for image interpretation of man-made scenes
Framework
25
![Page 26: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/26.jpg)
ETRIMS Database
Michael Yang 26
![Page 27: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/27.jpg)
One example image Ground truth labeling
Example Image
Michael Yang 27
![Page 28: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/28.jpg)
Region classifier (RDF) Pairwise CRF
Classification Results
Michael Yang 28
![Page 29: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/29.jpg)
HCRF Results
Image RDF CRF HCRF GT
Michael Yang 29
![Page 30: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/30.jpg)
Image GT
RDF CRF HCRF
HCRF Results
Michael Yang 30
![Page 31: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/31.jpg)
Pixelwise accuracy comparison
HCRF Results
Michael Yang 31
![Page 32: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/32.jpg)
4-connected CRF 8-connected CRF Fully-connected CRF
Fully Connected CRF
Michael Yang 32
![Page 33: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/33.jpg)
Fully Connected CRF
Michael Yang 33
Unary
Final
Image
Li, Yang, 2016
![Page 34: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/34.jpg)
Fully Connected CRF
Michael Yang 34
Image GT Texonboost CRF FC-CRF
![Page 35: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/35.jpg)
• Introduction
• Random Fields
Man-made Object Segmentation Semantic Video Segmentation
Outline
35
![Page 36: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/36.jpg)
Semantic Video Segmentation
Michael Yang
: :
![Page 37: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/37.jpg)
Deep Learning for Semantic Video Segmentation
Michael Yang
Badrinarayanan, Handa, Cipolla, arXiv 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic
Pixel-Wise Labelling
Semantic Video Segmentation
![Page 38: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/38.jpg)
Deep Learning
![Page 39: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/39.jpg)
Michael Yang
• Training CNN requires large amount of ground-truth data
• Dense labeling requires extensive human effort
• Labeling one image from CityScapes ~ 1.5 hours
Semantic Video Segmentation
![Page 40: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/40.jpg)
Semantic Video Segmentation
Michael Yang
• Use video to propagate labels. Pseudo Ground Truth (PGT)
Semantic Seg. Net (FCN)Train with
PGT
Mustikovela, Yang, Rother, ECCV Workshop 2016Can Ground Truth Label Propagation from Video help Semantic Segmentation?
![Page 41: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/41.jpg)
Semantic Video Segmentation
Michael Yang
• Use video to propagate labels. Pseudo Ground Truth (PGT)
Semantic Seg. Net (FCN)Train with
PGT
• Weakly-Supervised Learning CNN+CRF
Basic idea: given a few videos with limited labeled frames, we first estimate pseudo noisy ground truth for each frame in training set. Then we use all the labeled frames to train a CNN.
Mustikovela, Yang, Rother, ECCV Workshop 2016Can Ground Truth Label Propagation from Video help Semantic Segmentation?
![Page 42: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/42.jpg)
Semantic Video Segmentation
Generating Pseudo Ground Truth DataCRF for Label Propagation
Michael Yang
![Page 43: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/43.jpg)
Semantic Video Segmentation
Quality of Pseudo Ground Truth Data
Michael Yang
![Page 44: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/44.jpg)
Semantic Video Segmentation
CNN Training
Michael Yang
![Page 45: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/45.jpg)
Semantic Video Segmentation
Michael Yang
Different is good
More different but quality goes down
• Image 4 is more different to GT than Image 1• Quality of labeling of Image 5 might go down
CamVid Results
![Page 46: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/46.jpg)
Semantic Video Segmentation
Michael Yang
• Model trained with GT + 4th images performs the best• Performs better in 10/11 classes
CamVid Results
![Page 47: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/47.jpg)
Semantic Video Segmentation
Results
Video frame Our CNN result ground-truth
![Page 48: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/48.jpg)
Semantic Video Segmentation
Results
Video frame Our CNN result ground-truth
![Page 49: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/49.jpg)
• Introduction
• Random Fields
Man-made Object Segmentation Semantic Video Segmentation
Outline
49
![Page 50: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/50.jpg)
ITCUniversity of Twente, NL
Thank you!
![Page 51: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/51.jpg)
Image Labeling Problems
• Labelings highly structured
• Labels highly correlated with very complex dependencies
• Neighbouring pixels tend to take the same label
• Low number of connected components
• Classes present may be seen in one image
• Geometric / Location consistency
• Planarity in depth estimation
• … many others (task dependent)
51
![Page 52: Probabilistic Graphical Models - ITC · Semantic Video Segmentation Outline 3. ... probability theory & graph theory. 8. Bayesian networks. ... (Markov Random Field) 12](https://reader034.fdocuments.us/reader034/viewer/2022051801/5adff5127f8b9a97518ca885/html5/thumbnails/52.jpg)
Object-class Segmentation
Unary term Pairwise termUnary term
Pairwise term
where
Discriminatively trained classifier (RF, Boosting, etc.)
52