Junction Tree Algorithm
description
Transcript of Junction Tree Algorithm
![Page 1: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/1.jpg)
Junction Tree Algorithm
Brookes Vision Reading Group
![Page 2: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/2.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 3: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/3.jpg)
Don’t we all know …
• P(A) = 1 , if and only if A is certain
• P(A or B) = P(A)+P(B) if and only if A and B are mutually exclusive
• P(A,B) = P(A|B)P(B) = P(B|A)P(A)
• Conditional Independence– A is conditionally independent of C given B – P(A|B,C) = P(A|B)
![Page 4: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/4.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 5: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/5.jpg)
Graphical ModelsCompact graphical representation of joint probability.
A
B
P(A,B) = P(A)P(B|A)
A ‘causes’ B
![Page 6: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/6.jpg)
Graphical ModelsCompact graphical representation of joint probability.
A
B
P(A,B) = P(B)P(A|B)
B ‘causes’ A
![Page 7: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/7.jpg)
Graphical ModelsCompact graphical representation of joint probability.
A
B
P(A,B)
![Page 8: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/8.jpg)
A Simple Example
P(A,B,C) = P(A)P(B,C | A)
= P(A) P(B|A) P(C|B,A)
C is conditionally independent of A given B
= P(A) P(B|A) P(C|B)
Graphical Representation ???
![Page 9: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/9.jpg)
Bayesian NetworkDirected Graphical Model
P(U) = P(Vi | Pa(Vi))A
B
C
P(A,B,C) = P(A) P(B | A) P(C | B)
![Page 10: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/10.jpg)
Markov Random FieldsUndirected Graphical Model
A B C
![Page 11: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/11.jpg)
Markov Random FieldsUndirected Graphical Model
AB BCB
P(U) = P(Clique) / P(Separator)
Clique CliqueSeparator
P(A,B,C) = P(A,B) P(B,C) / P(B)
![Page 12: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/12.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 13: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/13.jpg)
Bayesian Networks
• A is conditionally independent of B given C
• Bayes ball cannot reach A from B
![Page 14: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/14.jpg)
Markov Random Fields
• A, B, C - (set of) nodes
• C is conditionally independent of A given B
• All paths from A to C go through B
![Page 15: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/15.jpg)
Markov Random Fields
![Page 16: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/16.jpg)
Markov Random Fields
A node is conditionally independent of all others given its neighbours.
![Page 17: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/17.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 18: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/18.jpg)
MAP Estimation
c*,s*,r*,w* = argmax P(C=c,S=s,R=r,W=w)
![Page 19: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/19.jpg)
Computing Marginals
P(W=w) = c,s,r P(C=c,S=s,R=r,W=w)
![Page 20: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/20.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 21: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/21.jpg)
Aim
• To perform exact inference efficiently
• Transform the graph into an appropriate data structure
• Ensure joint probability remains the same
• Ensure exact marginals can be computed
![Page 22: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/22.jpg)
Junction Tree Algorithm
• Converts Bayes Net into an undirected tree– Joint probability remains unchanged– Exact marginals can be computed
• Why ???– Uniform treatment of Bayes Net and MRF– Efficient inference is possible for undirected trees
![Page 23: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/23.jpg)
Junction Tree Algorithm
• Converts Bayes Net into an undirected tree– Joint probability remains unchanged– Exact marginals can be computed
• Why ???– Uniform treatment of Bayes Net and MRF– Efficient inference is possible for undirected trees
![Page 24: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/24.jpg)
Let us recap .. Shall we
P(U) = P(Vi | Pa(Vi))
= a(Vi , Pa(Vi)) Potential
Lets convert this to an undirected graphical model
A
B C
D
![Page 25: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/25.jpg)
Let us recap .. Shall weA
B C
D
Wait a second …something is wrong here.
The cliques of this graph are inconsistent with the original one.
Node D just lost a parent.
![Page 26: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/26.jpg)
SolutionA
B C
D
Ensure that a node and its parents are part of the same clique
Marry the parents for a happy family
Now you can make the graph undirected
![Page 27: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/27.jpg)
SolutionA
B C
D
A few conditional independences are lost.
But we have added extra edges, haven’t we ???
![Page 28: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/28.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 29: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/29.jpg)
Moralizing a graph
• Marry all unconnected parents
• Drop the edge directions
Ensure joint probability remains the same.
![Page 30: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/30.jpg)
Moralizing a graph
![Page 31: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/31.jpg)
Moralizing a graphC
S R
W
CSR
SRW
SR
Clique Potentials a(Ci)
Separator Potentials a(Si)
Initialize
a(Ci) = 1
a(Si) = 1
![Page 32: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/32.jpg)
Moralizing a graphC
S R
W
CSR
SRW
SR
Choose one node Vi
Find one clique Ci containing Vi and Pa(Vi)
Multiply a(Vi,Pa(Vi)) to a(Ci)
Repeat for all Vi
![Page 33: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/33.jpg)
Moralizing a graphC
S R
W
CSR
SRW
SR
Choose one node Vi
Find one clique Ci containing Vi and Pa(Vi)
Multiply a(Vi,Pa(Vi)) to a(Ci)
Repeat for all Vi
![Page 34: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/34.jpg)
Moralizing a graphC
S R
W
CSR
SRW
SR
Choose one node Vi
Find one clique Ci containing Vi and Pa(Vi)
Multiply a(Vi,Pa(Vi)) to a(Ci)
Repeat for all Vi
![Page 35: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/35.jpg)
Moralizing a graphC
S R
W
CSR
SRW
SR
Choose one node Vi
Find one clique Ci containing Vi and Pa(Vi)
Multiply a(Vi,Pa(Vi)) to a(Ci)
Repeat for all Vi
![Page 36: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/36.jpg)
Moralizing a graph
P(U) = a(Ci) / a(Si)
Now we can form a tree with all the cliques we chose.
That was easy. We’re ready to marginalize.
OR ARE WE ???
![Page 37: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/37.jpg)
A few more examples …
A B
DC
A B
C D
![Page 38: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/38.jpg)
A few more examples …
A B
DC
A B
C D
![Page 39: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/39.jpg)
A few more examples …
AB BD
CDAC
AB BCD
Inconsistency in C
Clearly we’re missing something here
![Page 40: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/40.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 41: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/41.jpg)
Junction Tree Property
In a junction tree, all cliques in the unique path between cliques Ci and Cj must contain CiCj
So what we want is a junction tree, right ???
Q. Do all graphs have a junction tree ???
A. NO
![Page 42: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/42.jpg)
Decomposable GraphsDecomposition (A,B,C)
A C B
• V = A B C
• All paths between A and B go through C
• C is a complete subset of V
Undirected graph G = (V,E)
![Page 43: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/43.jpg)
Decomposable Graphs
• A, B and/or C can be empty
• A, B are non-empty in a proper decomposition
![Page 44: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/44.jpg)
Decomposable Graphs
• G is decomposable if and only if
• G is complete OR
• It possesses a proper decomposition (A,B,C) such that– GAC is decomposable
– GBC is decomposable
![Page 45: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/45.jpg)
Decomposable Graphs
A B
DC
A B
C D
Not Decomposable Decomposable
![Page 46: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/46.jpg)
Decomposable Graphs
Not Decomposable Decomposable
A
B C
ED
A
B C
ED
![Page 47: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/47.jpg)
An Important Theorem
Theorem: A graph G has a junction tree if and only if it is decomposable.
Proof on white board.
![Page 48: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/48.jpg)
OK. So how do I convert my graph into a decomposable one.
![Page 49: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/49.jpg)
Time for more definitions
• Chord of a cycle– An edge between two non-successive nodes
• Chordless cycle– A cycle with no chords
• Triangulated graph– A graph with no chordless cycles
![Page 50: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/50.jpg)
Another Important Theorem
Theorem: A graph G is decomposableif and only if it is triangulated.
Proof on white board.
Alright. So add edges to triangulate the graph.
![Page 51: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/51.jpg)
Triangulating a Graph
A B
C D
ABC BCDBC
![Page 52: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/52.jpg)
Triangulating a Graph
![Page 53: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/53.jpg)
Triangulating a Graph
![Page 54: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/54.jpg)
Some Notes on Triangulation
Can we ensure the joint probability remains unchanged ??
Of course. Adding edges preserves cliques found after moralization.
Use the previous algorithm for initializing potentials.
Aren’t more conditional independences lost ???
Yes. :-(
![Page 55: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/55.jpg)
Some Notes on TriangulationIs Triangulation unique??
No.
Okay then. Lets find the best triangulation.
Sadly, that’s NP hard.
Hang on. We still have a graph. We were promised a tree.
Alright. Lets form a tree then.
![Page 56: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/56.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 57: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/57.jpg)
Creating a Junction Tree
A
B D
C E
ABD
CDE
BCD
ABD
BCD
CDE
Not a junction tree
Junction tree
Clearly, we’re still missing something here.
![Page 58: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/58.jpg)
Yet Another Theorem
Theorem: A junction tree is an MSTwhere the weights are the cardinalityof the separators.
Proof on white board.
Alright. So lets form an MST.
![Page 59: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/59.jpg)
A
B D
C E
Forming an MST
ABD
BCD CDE
2
2
1
![Page 60: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/60.jpg)
A
B D
C E
Forming an MST
ABD
BCD CDE
2
2
1
![Page 61: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/61.jpg)
A
B D
C E
Forming an MST
ABD
BCD CDE
2
2
1
![Page 62: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/62.jpg)
A
B D
C E
Forming an MST
ABD
BCD CDE
2
2
![Page 63: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/63.jpg)
A Quick Recap
A S
BLT
E
X D
Asia Network
![Page 64: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/64.jpg)
A Quick Recap
A S
BLT
E
X D
1. Marry unconnected parents
![Page 65: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/65.jpg)
A Quick Recap
A S
BLT
E
X D
2. Drop directionality of edges.
![Page 66: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/66.jpg)
A Quick Recap
A S
BLT
E
X D
3. Triangulate the graph.
![Page 67: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/67.jpg)
A Quick Recap4. Find the MST clique tree. Voila .. The junction tree.
SBL
BLE
DBEXE
TLEAT
Whew. Done !!
But where are these marginals we were talking about ?
![Page 68: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/68.jpg)
Outline• Graphical Models
– What are Graphical Models ?– Conditional Independence– Inference
• Junction Tree Algorithm– Moralizing a graph– Junction Tree Property– Creating a junction tree– Inference using junction tree algorithm
![Page 69: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/69.jpg)
Inference using JTA
• Modify potentials
• Ensure joint probability is consistent
• Ensure consistency between neighbouring cliques
• Ensure clique potentials = clique marginals
• Ensure separator potentials = separator marginals
![Page 70: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/70.jpg)
Inference using JTA
V WS
1. a*(S) = V\S a(V)
2. a*(W) = a(W) a*(S) / a(S)
3. a**(S) = W\S a*(W)
4. a*(V) = a(V) a**(S) / a*(S)
V\S a*(V) = a**(S)
= W\S a*(W)
Consistency
![Page 71: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/71.jpg)
Inference using JTA
V WS
1. a*(S) = V\S a(V)
2. a*(W) = a(W) a*(S) / a(S)
3. a**(S) = W\S a*(W)
4. a*(V) = a(V) a**(S) / a*(S)
a*(V) a*(W) / a**(S)
= a(V) a(W) / a(S)
Joint probabilityremains same
![Page 72: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/72.jpg)
One Last Theorem
Theorem: After JTA, Potentials = Marginals
Proof on white board.
(Then we can all go home)
![Page 73: Junction Tree Algorithm](https://reader035.fdocuments.us/reader035/viewer/2022062322/568150a7550346895dbeb307/html5/thumbnails/73.jpg)
Happy Marginalizing